2023-03-29 03:53:32,544 DEBUG [main] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/3541dc51-7937-e96e-6c09-468c657fb2cc 2023-03-29 03:53:32,557 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.TestAcidGuaranteesWithEagerPolicy timeout: 13 mins 2023-03-29 03:53:32,574 INFO [Time-limited test] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=1, rsPorts=, rsClass=null, numDataNodes=1, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-03-29 03:53:32,575 INFO [Time-limited test] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/3541dc51-7937-e96e-6c09-468c657fb2cc/cluster_c6f3e91d-ff3e-c1b8-1e7d-cfb1945e516d, deleteOnExit=true 2023-03-29 03:53:32,575 INFO [Time-limited test] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-03-29 03:53:32,575 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/3541dc51-7937-e96e-6c09-468c657fb2cc/test.cache.data in system properties and HBase conf 2023-03-29 03:53:32,576 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/3541dc51-7937-e96e-6c09-468c657fb2cc/hadoop.tmp.dir in system properties and HBase conf 2023-03-29 03:53:32,576 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/3541dc51-7937-e96e-6c09-468c657fb2cc/hadoop.log.dir in system properties and HBase conf 2023-03-29 03:53:32,577 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/3541dc51-7937-e96e-6c09-468c657fb2cc/mapreduce.cluster.local.dir in system properties and HBase conf 2023-03-29 03:53:32,577 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/3541dc51-7937-e96e-6c09-468c657fb2cc/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-03-29 03:53:32,577 INFO [Time-limited test] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-03-29 03:53:32,698 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2023-03-29 03:53:33,111 DEBUG [Time-limited test] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-03-29 03:53:33,115 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/3541dc51-7937-e96e-6c09-468c657fb2cc/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-03-29 03:53:33,116 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/3541dc51-7937-e96e-6c09-468c657fb2cc/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-03-29 03:53:33,116 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/3541dc51-7937-e96e-6c09-468c657fb2cc/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-03-29 03:53:33,116 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/3541dc51-7937-e96e-6c09-468c657fb2cc/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-03-29 03:53:33,116 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/3541dc51-7937-e96e-6c09-468c657fb2cc/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-03-29 03:53:33,117 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/3541dc51-7937-e96e-6c09-468c657fb2cc/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-03-29 03:53:33,117 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/3541dc51-7937-e96e-6c09-468c657fb2cc/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-03-29 03:53:33,117 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/3541dc51-7937-e96e-6c09-468c657fb2cc/dfs.journalnode.edits.dir in system properties and HBase conf 2023-03-29 03:53:33,118 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/3541dc51-7937-e96e-6c09-468c657fb2cc/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-03-29 03:53:33,118 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/3541dc51-7937-e96e-6c09-468c657fb2cc/nfs.dump.dir in system properties and HBase conf 2023-03-29 03:53:33,119 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/3541dc51-7937-e96e-6c09-468c657fb2cc/java.io.tmpdir in system properties and HBase conf 2023-03-29 03:53:33,119 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/3541dc51-7937-e96e-6c09-468c657fb2cc/dfs.journalnode.edits.dir in system properties and HBase conf 2023-03-29 03:53:33,119 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/3541dc51-7937-e96e-6c09-468c657fb2cc/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-03-29 03:53:33,119 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/3541dc51-7937-e96e-6c09-468c657fb2cc/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-03-29 03:53:33,698 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-03-29 03:53:33,702 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-03-29 03:53:34,023 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2023-03-29 03:53:34,200 INFO [Time-limited test] log.Slf4jLog(67): Logging to org.slf4j.impl.Reload4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2023-03-29 03:53:34,221 WARN [Time-limited test] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-03-29 03:53:34,264 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2023-03-29 03:53:34,295 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/3541dc51-7937-e96e-6c09-468c657fb2cc/java.io.tmpdir/Jetty_localhost_36655_hdfs____2bi30d/webapp 2023-03-29 03:53:34,455 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:36655 2023-03-29 03:53:34,542 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-03-29 03:53:34,542 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-03-29 03:53:34,999 WARN [Listener at localhost/42373] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-03-29 03:53:35,163 WARN [Listener at localhost/42373] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-03-29 03:53:35,186 WARN [Listener at localhost/42373] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-03-29 03:53:35,195 INFO [Listener at localhost/42373] log.Slf4jLog(67): jetty-6.1.26 2023-03-29 03:53:35,202 INFO [Listener at localhost/42373] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/3541dc51-7937-e96e-6c09-468c657fb2cc/java.io.tmpdir/Jetty_localhost_43771_datanode____rxa1cp/webapp 2023-03-29 03:53:35,337 INFO [Listener at localhost/42373] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:43771 2023-03-29 03:53:35,816 WARN [Listener at localhost/35027] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-03-29 03:53:36,346 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xc94ba7615a02fde2: Processing first storage report for DS-5c7b0cb0-3cf0-4872-a281-2cda2982633f from datanode ff5b43e4-08a3-49e1-bbac-436ca110ab7a 2023-03-29 03:53:36,348 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xc94ba7615a02fde2: from storage DS-5c7b0cb0-3cf0-4872-a281-2cda2982633f node DatanodeRegistration(127.0.0.1:34435, datanodeUuid=ff5b43e4-08a3-49e1-bbac-436ca110ab7a, infoPort=38667, infoSecurePort=0, ipcPort=35027, storageInfo=lv=-57;cid=testClusterID;nsid=1928977802;c=1680062013777), blocks: 0, hasStaleStorage: true, processing time: 2 msecs, invalidatedBlocks: 0 2023-03-29 03:53:36,348 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xc94ba7615a02fde2: Processing first storage report for DS-536258df-fa93-466e-9c6f-a8556e649694 from datanode ff5b43e4-08a3-49e1-bbac-436ca110ab7a 2023-03-29 03:53:36,348 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xc94ba7615a02fde2: from storage DS-536258df-fa93-466e-9c6f-a8556e649694 node DatanodeRegistration(127.0.0.1:34435, datanodeUuid=ff5b43e4-08a3-49e1-bbac-436ca110ab7a, infoPort=38667, infoSecurePort=0, ipcPort=35027, storageInfo=lv=-57;cid=testClusterID;nsid=1928977802;c=1680062013777), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-03-29 03:53:36,378 DEBUG [Listener at localhost/35027] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/3541dc51-7937-e96e-6c09-468c657fb2cc 2023-03-29 03:53:36,457 INFO [Listener at localhost/35027] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/3541dc51-7937-e96e-6c09-468c657fb2cc/cluster_c6f3e91d-ff3e-c1b8-1e7d-cfb1945e516d/zookeeper_0, clientPort=52103, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/3541dc51-7937-e96e-6c09-468c657fb2cc/cluster_c6f3e91d-ff3e-c1b8-1e7d-cfb1945e516d/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/3541dc51-7937-e96e-6c09-468c657fb2cc/cluster_c6f3e91d-ff3e-c1b8-1e7d-cfb1945e516d/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-03-29 03:53:36,471 INFO [Listener at localhost/35027] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=52103 2023-03-29 03:53:36,481 INFO [Listener at localhost/35027] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-03-29 03:53:36,484 INFO [Listener at localhost/35027] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-03-29 03:53:37,163 INFO [Listener at localhost/35027] util.FSUtils(471): Created version file at hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f with version=8 2023-03-29 03:53:37,163 INFO [Listener at localhost/35027] hbase.HBaseTestingUtility(1406): Setting hbase.fs.tmp.dir to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/hbase-staging 2023-03-29 03:53:37,473 INFO [Listener at localhost/35027] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-03-29 03:53:37,983 INFO [Listener at localhost/35027] client.ConnectionUtils(127): master/jenkins-hbase4:0 server-side Connection retries=45 2023-03-29 03:53:38,015 INFO [Listener at localhost/35027] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-03-29 03:53:38,016 INFO [Listener at localhost/35027] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-03-29 03:53:38,016 INFO [Listener at localhost/35027] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-03-29 03:53:38,016 INFO [Listener at localhost/35027] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-03-29 03:53:38,016 INFO [Listener at localhost/35027] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-03-29 03:53:38,156 INFO [Listener at localhost/35027] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-03-29 03:53:38,234 DEBUG [Listener at localhost/35027] util.ClassSize(228): Using Unsafe to estimate memory layout 2023-03-29 03:53:38,331 INFO [Listener at localhost/35027] ipc.NettyRpcServer(120): Bind to /172.31.14.131:43505 2023-03-29 03:53:38,341 INFO [Listener at localhost/35027] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-03-29 03:53:38,344 INFO [Listener at localhost/35027] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-03-29 03:53:38,366 INFO [Listener at localhost/35027] zookeeper.RecoverableZooKeeper(93): Process identifier=master:43505 connecting to ZooKeeper ensemble=127.0.0.1:52103 2023-03-29 03:53:38,406 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:435050x0, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-03-29 03:53:38,409 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): master:43505-0x1012e67d5710000 connected 2023-03-29 03:53:38,437 DEBUG [Listener at localhost/35027] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-03-29 03:53:38,438 DEBUG [Listener at localhost/35027] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-03-29 03:53:38,441 DEBUG [Listener at localhost/35027] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-03-29 03:53:38,449 DEBUG [Listener at localhost/35027] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=43505 2023-03-29 03:53:38,449 DEBUG [Listener at localhost/35027] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=43505 2023-03-29 03:53:38,449 DEBUG [Listener at localhost/35027] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=43505 2023-03-29 03:53:38,450 DEBUG [Listener at localhost/35027] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=43505 2023-03-29 03:53:38,450 DEBUG [Listener at localhost/35027] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=43505 2023-03-29 03:53:38,458 INFO [Listener at localhost/35027] master.HMaster(444): hbase.rootdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f, hbase.cluster.distributed=false 2023-03-29 03:53:38,535 INFO [Listener at localhost/35027] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-03-29 03:53:38,535 INFO [Listener at localhost/35027] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-03-29 03:53:38,535 INFO [Listener at localhost/35027] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-03-29 03:53:38,535 INFO [Listener at localhost/35027] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-03-29 03:53:38,536 INFO [Listener at localhost/35027] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-03-29 03:53:38,536 INFO [Listener at localhost/35027] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-03-29 03:53:38,541 INFO [Listener at localhost/35027] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-03-29 03:53:38,543 INFO [Listener at localhost/35027] ipc.NettyRpcServer(120): Bind to /172.31.14.131:35599 2023-03-29 03:53:38,545 INFO [Listener at localhost/35027] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-03-29 03:53:38,552 DEBUG [Listener at localhost/35027] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-03-29 03:53:38,553 INFO [Listener at localhost/35027] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-03-29 03:53:38,556 INFO [Listener at localhost/35027] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-03-29 03:53:38,559 INFO [Listener at localhost/35027] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:35599 connecting to ZooKeeper ensemble=127.0.0.1:52103 2023-03-29 03:53:38,562 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:355990x0, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-03-29 03:53:38,563 DEBUG [Listener at localhost/35027] zookeeper.ZKUtil(164): regionserver:355990x0, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-03-29 03:53:38,564 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:35599-0x1012e67d5710001 connected 2023-03-29 03:53:38,565 DEBUG [Listener at localhost/35027] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-03-29 03:53:38,566 DEBUG [Listener at localhost/35027] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-03-29 03:53:38,567 DEBUG [Listener at localhost/35027] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=35599 2023-03-29 03:53:38,568 DEBUG [Listener at localhost/35027] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=35599 2023-03-29 03:53:38,568 DEBUG [Listener at localhost/35027] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=35599 2023-03-29 03:53:38,569 DEBUG [Listener at localhost/35027] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=35599 2023-03-29 03:53:38,569 DEBUG [Listener at localhost/35027] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=35599 2023-03-29 03:53:38,571 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase4.apache.org,43505,1680062017306 2023-03-29 03:53:38,581 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-03-29 03:53:38,582 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase4.apache.org,43505,1680062017306 2023-03-29 03:53:38,602 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-03-29 03:53:38,602 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-03-29 03:53:38,603 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-29 03:53:38,603 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-03-29 03:53:38,605 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase4.apache.org,43505,1680062017306 from backup master directory 2023-03-29 03:53:38,605 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-03-29 03:53:38,608 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase4.apache.org,43505,1680062017306 2023-03-29 03:53:38,608 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-03-29 03:53:38,609 WARN [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-03-29 03:53:38,609 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase4.apache.org,43505,1680062017306 2023-03-29 03:53:38,611 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating data MemStoreChunkPool with chunk size 2 MB, max count 352, initial count 0 2023-03-29 03:53:38,613 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 391, initial count 0 2023-03-29 03:53:38,701 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] util.FSUtils(620): Created cluster ID file at hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/hbase.id with ID: a18b760f-89c6-4379-817d-c7ec30350239 2023-03-29 03:53:38,746 INFO [master/jenkins-hbase4:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-03-29 03:53:38,760 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-29 03:53:38,804 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x5465106c to 127.0.0.1:52103 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-29 03:53:38,839 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2694ce81, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-29 03:53:38,864 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-29 03:53:38,865 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-03-29 03:53:38,882 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(264): ClientProtocol::create wrong number of arguments, should be hadoop 3.2 or below 2023-03-29 03:53:38,883 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(270): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2023-03-29 03:53:38,885 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(279): can not find SHOULD_REPLICATE flag, should be hadoop 2.x java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.fs.CreateFlag.SHOULD_REPLICATE at java.lang.Enum.valueOf(Enum.java:238) at org.apache.hadoop.fs.CreateFlag.valueOf(CreateFlag.java:63) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.loadShouldReplicateFlag(FanOutOneBlockAsyncDFSOutputHelper.java:277) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:304) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:139) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-03-29 03:53:38,888 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(243): No decryptEncryptedDataEncryptionKey method in DFSClient, should be hadoop version with HDFS-12396 java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo) at java.lang.Class.getDeclaredMethod(Class.java:2130) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelperWithoutHDFS12396(FanOutOneBlockAsyncDFSOutputSaslHelper.java:182) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:241) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:252) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:140) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-03-29 03:53:38,890 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-03-29 03:53:38,923 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7693): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/MasterData/data/master/store-tmp 2023-03-29 03:53:38,954 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-29 03:53:38,955 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-03-29 03:53:38,955 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-29 03:53:38,955 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-29 03:53:38,955 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-03-29 03:53:38,955 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-29 03:53:38,955 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-29 03:53:38,955 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-03-29 03:53:38,957 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/MasterData/WALs/jenkins-hbase4.apache.org,43505,1680062017306 2023-03-29 03:53:38,981 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C43505%2C1680062017306, suffix=, logDir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/MasterData/WALs/jenkins-hbase4.apache.org,43505,1680062017306, archiveDir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/MasterData/oldWALs, maxLogs=10 2023-03-29 03:53:39,047 DEBUG [RS-EventLoopGroup-3-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:34435,DS-5c7b0cb0-3cf0-4872-a281-2cda2982633f,DISK] 2023-03-29 03:53:39,054 DEBUG [RS-EventLoopGroup-3-2] asyncfs.ProtobufDecoder(123): Hadoop 3.2 and below use unshaded protobuf. java.lang.ClassNotFoundException: org.apache.hadoop.thirdparty.protobuf.MessageLite at java.net.URLClassLoader.findClass(URLClassLoader.java:387) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.ProtobufDecoder.(ProtobufDecoder.java:118) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.processWriteBlockResponse(FanOutOneBlockAsyncDFSOutputHelper.java:340) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$100(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$4.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:424) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:185) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:418) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:476) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:471) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:653) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:691) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-29 03:53:39,129 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/MasterData/WALs/jenkins-hbase4.apache.org,43505,1680062017306/jenkins-hbase4.apache.org%2C43505%2C1680062017306.1680062018992 2023-03-29 03:53:39,130 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:34435,DS-5c7b0cb0-3cf0-4872-a281-2cda2982633f,DISK]] 2023-03-29 03:53:39,131 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-03-29 03:53:39,131 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-29 03:53:39,135 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-03-29 03:53:39,136 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-03-29 03:53:39,203 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-03-29 03:53:39,210 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-03-29 03:53:39,239 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-03-29 03:53:39,254 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-29 03:53:39,261 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-03-29 03:53:39,263 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-03-29 03:53:39,282 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-03-29 03:53:39,287 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-29 03:53:39,288 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60091217, jitterRate=-0.10457108914852142}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-03-29 03:53:39,289 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-03-29 03:53:39,290 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-03-29 03:53:39,312 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-03-29 03:53:39,312 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-03-29 03:53:39,314 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-03-29 03:53:39,316 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 1 msec 2023-03-29 03:53:39,349 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 32 msec 2023-03-29 03:53:39,349 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(95): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-03-29 03:53:39,378 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-03-29 03:53:39,383 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-03-29 03:53:39,407 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-03-29 03:53:39,411 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-03-29 03:53:39,413 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-03-29 03:53:39,418 INFO [master/jenkins-hbase4:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-03-29 03:53:39,422 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-03-29 03:53:39,425 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-29 03:53:39,427 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-03-29 03:53:39,427 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-03-29 03:53:39,440 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-03-29 03:53:39,445 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-03-29 03:53:39,445 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-03-29 03:53:39,445 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-29 03:53:39,446 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase4.apache.org,43505,1680062017306, sessionid=0x1012e67d5710000, setting cluster-up flag (Was=false) 2023-03-29 03:53:39,461 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-29 03:53:39,468 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-03-29 03:53:39,470 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,43505,1680062017306 2023-03-29 03:53:39,476 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-29 03:53:39,483 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-03-29 03:53:39,484 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,43505,1680062017306 2023-03-29 03:53:39,487 WARN [master/jenkins-hbase4:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/.hbase-snapshot/.tmp 2023-03-29 03:53:39,574 INFO [RS:0;jenkins-hbase4:35599] regionserver.HRegionServer(952): ClusterId : a18b760f-89c6-4379-817d-c7ec30350239 2023-03-29 03:53:39,579 DEBUG [RS:0;jenkins-hbase4:35599] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-03-29 03:53:39,586 DEBUG [RS:0;jenkins-hbase4:35599] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-03-29 03:53:39,586 DEBUG [RS:0;jenkins-hbase4:35599] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-03-29 03:53:39,593 DEBUG [RS:0;jenkins-hbase4:35599] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-03-29 03:53:39,594 DEBUG [RS:0;jenkins-hbase4:35599] zookeeper.ReadOnlyZKClient(139): Connect 0x1a0467bd to 127.0.0.1:52103 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-29 03:53:39,601 DEBUG [RS:0;jenkins-hbase4:35599] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@14c39a8d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-29 03:53:39,601 DEBUG [RS:0;jenkins-hbase4:35599] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@10fc0ccd, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-03-29 03:53:39,611 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-03-29 03:53:39,624 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-03-29 03:53:39,624 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-03-29 03:53:39,625 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-03-29 03:53:39,625 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-03-29 03:53:39,625 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase4:0, corePoolSize=10, maxPoolSize=10 2023-03-29 03:53:39,625 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-03-29 03:53:39,625 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-03-29 03:53:39,625 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-03-29 03:53:39,628 DEBUG [RS:0;jenkins-hbase4:35599] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase4:35599 2023-03-29 03:53:39,632 INFO [RS:0;jenkins-hbase4:35599] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-03-29 03:53:39,632 INFO [RS:0;jenkins-hbase4:35599] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-03-29 03:53:39,632 DEBUG [RS:0;jenkins-hbase4:35599] regionserver.HRegionServer(1023): About to register with Master. 2023-03-29 03:53:39,634 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1680062049634 2023-03-29 03:53:39,637 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-03-29 03:53:39,637 INFO [RS:0;jenkins-hbase4:35599] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,43505,1680062017306 with isa=jenkins-hbase4.apache.org/172.31.14.131:35599, startcode=1680062018534 2023-03-29 03:53:39,639 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-03-29 03:53:39,640 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-03-29 03:53:39,646 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-03-29 03:53:39,650 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-03-29 03:53:39,656 DEBUG [RS:0;jenkins-hbase4:35599] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-03-29 03:53:39,659 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-03-29 03:53:39,660 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-03-29 03:53:39,660 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-03-29 03:53:39,660 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-03-29 03:53:39,661 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-03-29 03:53:39,662 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-03-29 03:53:39,665 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-03-29 03:53:39,665 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-03-29 03:53:39,669 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-03-29 03:53:39,669 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-03-29 03:53:39,671 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1680062019671,5,FailOnTimeoutGroup] 2023-03-29 03:53:39,672 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1680062019672,5,FailOnTimeoutGroup] 2023-03-29 03:53:39,672 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-03-29 03:53:39,672 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-03-29 03:53:39,674 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-03-29 03:53:39,674 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-03-29 03:53:39,702 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-03-29 03:53:39,703 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-03-29 03:53:39,703 INFO [PEWorker-1] regionserver.HRegion(7675): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f 2023-03-29 03:53:39,719 DEBUG [PEWorker-1] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-29 03:53:39,723 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-03-29 03:53:39,727 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/hbase/meta/1588230740/info 2023-03-29 03:53:39,730 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-03-29 03:53:39,731 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-29 03:53:39,732 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-03-29 03:53:39,739 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:54233, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.0 (auth:SIMPLE), service=RegionServerStatusService 2023-03-29 03:53:39,742 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/hbase/meta/1588230740/rep_barrier 2023-03-29 03:53:39,743 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-03-29 03:53:39,747 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-29 03:53:39,747 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-03-29 03:53:39,750 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/hbase/meta/1588230740/table 2023-03-29 03:53:39,751 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:39,752 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-03-29 03:53:39,753 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-29 03:53:39,756 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/hbase/meta/1588230740 2023-03-29 03:53:39,757 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/hbase/meta/1588230740 2023-03-29 03:53:39,761 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-29 03:53:39,764 DEBUG [PEWorker-1] regionserver.HRegion(1055): writing seq id for 1588230740 2023-03-29 03:53:39,770 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-29 03:53:39,773 DEBUG [RS:0;jenkins-hbase4:35599] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f 2023-03-29 03:53:39,774 DEBUG [RS:0;jenkins-hbase4:35599] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:42373 2023-03-29 03:53:39,774 DEBUG [RS:0;jenkins-hbase4:35599] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-03-29 03:53:39,774 INFO [PEWorker-1] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72680135, jitterRate=0.08301840722560883}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-29 03:53:39,774 DEBUG [PEWorker-1] regionserver.HRegion(965): Region open journal for 1588230740: 2023-03-29 03:53:39,775 DEBUG [PEWorker-1] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-03-29 03:53:39,775 INFO [PEWorker-1] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-03-29 03:53:39,775 DEBUG [PEWorker-1] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-03-29 03:53:39,775 DEBUG [PEWorker-1] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-03-29 03:53:39,775 DEBUG [PEWorker-1] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-03-29 03:53:39,776 INFO [PEWorker-1] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-03-29 03:53:39,776 DEBUG [PEWorker-1] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-03-29 03:53:39,780 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-03-29 03:53:39,780 DEBUG [RS:0;jenkins-hbase4:35599] zookeeper.ZKUtil(162): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:39,781 WARN [RS:0;jenkins-hbase4:35599] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-03-29 03:53:39,781 INFO [RS:0;jenkins-hbase4:35599] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-03-29 03:53:39,782 DEBUG [RS:0;jenkins-hbase4:35599] regionserver.HRegionServer(1947): logDir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/WALs/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:39,785 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-03-29 03:53:39,785 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-03-29 03:53:39,786 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,35599,1680062018534] 2023-03-29 03:53:39,794 DEBUG [RS:0;jenkins-hbase4:35599] zookeeper.ZKUtil(162): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:39,796 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-03-29 03:53:39,807 DEBUG [RS:0;jenkins-hbase4:35599] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-03-29 03:53:39,811 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-03-29 03:53:39,817 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-03-29 03:53:39,819 INFO [RS:0;jenkins-hbase4:35599] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-03-29 03:53:39,844 INFO [RS:0;jenkins-hbase4:35599] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-03-29 03:53:39,848 INFO [RS:0;jenkins-hbase4:35599] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-03-29 03:53:39,848 INFO [RS:0;jenkins-hbase4:35599] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-03-29 03:53:39,849 INFO [RS:0;jenkins-hbase4:35599] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-03-29 03:53:39,857 INFO [RS:0;jenkins-hbase4:35599] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-03-29 03:53:39,857 DEBUG [RS:0;jenkins-hbase4:35599] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-03-29 03:53:39,857 DEBUG [RS:0;jenkins-hbase4:35599] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-03-29 03:53:39,858 DEBUG [RS:0;jenkins-hbase4:35599] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-03-29 03:53:39,858 DEBUG [RS:0;jenkins-hbase4:35599] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-03-29 03:53:39,858 DEBUG [RS:0;jenkins-hbase4:35599] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-03-29 03:53:39,858 DEBUG [RS:0;jenkins-hbase4:35599] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-03-29 03:53:39,858 DEBUG [RS:0;jenkins-hbase4:35599] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-03-29 03:53:39,858 DEBUG [RS:0;jenkins-hbase4:35599] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-03-29 03:53:39,858 DEBUG [RS:0;jenkins-hbase4:35599] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-03-29 03:53:39,858 DEBUG [RS:0;jenkins-hbase4:35599] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-03-29 03:53:39,859 INFO [RS:0;jenkins-hbase4:35599] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-03-29 03:53:39,859 INFO [RS:0;jenkins-hbase4:35599] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-03-29 03:53:39,859 INFO [RS:0;jenkins-hbase4:35599] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-03-29 03:53:39,876 INFO [RS:0;jenkins-hbase4:35599] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-03-29 03:53:39,879 INFO [RS:0;jenkins-hbase4:35599] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,35599,1680062018534-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-03-29 03:53:39,896 INFO [RS:0;jenkins-hbase4:35599] regionserver.Replication(203): jenkins-hbase4.apache.org,35599,1680062018534 started 2023-03-29 03:53:39,897 INFO [RS:0;jenkins-hbase4:35599] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,35599,1680062018534, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:35599, sessionid=0x1012e67d5710001 2023-03-29 03:53:39,897 DEBUG [RS:0;jenkins-hbase4:35599] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-03-29 03:53:39,897 DEBUG [RS:0;jenkins-hbase4:35599] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:39,897 DEBUG [RS:0;jenkins-hbase4:35599] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,35599,1680062018534' 2023-03-29 03:53:39,897 DEBUG [RS:0;jenkins-hbase4:35599] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:53:39,898 DEBUG [RS:0;jenkins-hbase4:35599] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:53:39,898 DEBUG [RS:0;jenkins-hbase4:35599] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-03-29 03:53:39,899 DEBUG [RS:0;jenkins-hbase4:35599] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-03-29 03:53:39,899 DEBUG [RS:0;jenkins-hbase4:35599] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:39,899 DEBUG [RS:0;jenkins-hbase4:35599] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,35599,1680062018534' 2023-03-29 03:53:39,899 DEBUG [RS:0;jenkins-hbase4:35599] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-03-29 03:53:39,899 DEBUG [RS:0;jenkins-hbase4:35599] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-03-29 03:53:39,900 DEBUG [RS:0;jenkins-hbase4:35599] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-03-29 03:53:39,900 INFO [RS:0;jenkins-hbase4:35599] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-03-29 03:53:39,900 INFO [RS:0;jenkins-hbase4:35599] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-03-29 03:53:39,969 DEBUG [jenkins-hbase4:43505] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-03-29 03:53:39,971 INFO [PEWorker-2] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,35599,1680062018534, state=OPENING 2023-03-29 03:53:39,979 DEBUG [PEWorker-2] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-03-29 03:53:39,981 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-29 03:53:39,982 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-03-29 03:53:39,985 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,35599,1680062018534}] 2023-03-29 03:53:40,011 INFO [RS:0;jenkins-hbase4:35599] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C35599%2C1680062018534, suffix=, logDir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/WALs/jenkins-hbase4.apache.org,35599,1680062018534, archiveDir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/oldWALs, maxLogs=32 2023-03-29 03:53:40,032 DEBUG [RS-EventLoopGroup-3-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:34435,DS-5c7b0cb0-3cf0-4872-a281-2cda2982633f,DISK] 2023-03-29 03:53:40,036 INFO [RS:0;jenkins-hbase4:35599] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/WALs/jenkins-hbase4.apache.org,35599,1680062018534/jenkins-hbase4.apache.org%2C35599%2C1680062018534.1680062020015 2023-03-29 03:53:40,036 DEBUG [RS:0;jenkins-hbase4:35599] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:34435,DS-5c7b0cb0-3cf0-4872-a281-2cda2982633f,DISK]] 2023-03-29 03:53:40,173 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:40,176 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-03-29 03:53:40,180 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:45076, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-03-29 03:53:40,191 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:meta,,1.1588230740 2023-03-29 03:53:40,192 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-03-29 03:53:40,195 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C35599%2C1680062018534.meta, suffix=.meta, logDir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/WALs/jenkins-hbase4.apache.org,35599,1680062018534, archiveDir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/oldWALs, maxLogs=32 2023-03-29 03:53:40,215 DEBUG [RS-EventLoopGroup-3-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:34435,DS-5c7b0cb0-3cf0-4872-a281-2cda2982633f,DISK] 2023-03-29 03:53:40,220 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/WALs/jenkins-hbase4.apache.org,35599,1680062018534/jenkins-hbase4.apache.org%2C35599%2C1680062018534.meta.1680062020198.meta 2023-03-29 03:53:40,224 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:34435,DS-5c7b0cb0-3cf0-4872-a281-2cda2982633f,DISK]] 2023-03-29 03:53:40,224 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-03-29 03:53:40,226 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-03-29 03:53:40,244 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-03-29 03:53:40,249 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-03-29 03:53:40,255 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-03-29 03:53:40,255 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-29 03:53:40,255 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-03-29 03:53:40,255 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-03-29 03:53:40,258 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-03-29 03:53:40,260 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/hbase/meta/1588230740/info 2023-03-29 03:53:40,260 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/hbase/meta/1588230740/info 2023-03-29 03:53:40,261 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-03-29 03:53:40,262 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-29 03:53:40,262 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-03-29 03:53:40,264 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/hbase/meta/1588230740/rep_barrier 2023-03-29 03:53:40,264 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/hbase/meta/1588230740/rep_barrier 2023-03-29 03:53:40,265 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-03-29 03:53:40,267 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-29 03:53:40,268 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-03-29 03:53:40,269 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/hbase/meta/1588230740/table 2023-03-29 03:53:40,269 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/hbase/meta/1588230740/table 2023-03-29 03:53:40,270 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-03-29 03:53:40,271 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-29 03:53:40,273 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/hbase/meta/1588230740 2023-03-29 03:53:40,276 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/hbase/meta/1588230740 2023-03-29 03:53:40,280 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-29 03:53:40,282 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-03-29 03:53:40,284 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65019409, jitterRate=-0.031135305762290955}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-29 03:53:40,284 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-03-29 03:53:40,297 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1680062020164 2023-03-29 03:53:40,317 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:meta,,1.1588230740 2023-03-29 03:53:40,318 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:meta,,1.1588230740 2023-03-29 03:53:40,318 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,35599,1680062018534, state=OPEN 2023-03-29 03:53:40,321 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-03-29 03:53:40,321 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-03-29 03:53:40,325 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-03-29 03:53:40,325 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,35599,1680062018534 in 336 msec 2023-03-29 03:53:40,330 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-03-29 03:53:40,330 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 530 msec 2023-03-29 03:53:40,336 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 799 msec 2023-03-29 03:53:40,337 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1680062020337, completionTime=-1 2023-03-29 03:53:40,337 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=1; waited=0ms, expected min=1 server(s), max=1 server(s), master is running 2023-03-29 03:53:40,337 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-03-29 03:53:40,408 DEBUG [hconnection-0xf5645e1-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-29 03:53:40,412 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:45078, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-29 03:53:40,430 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-03-29 03:53:40,430 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1680062080430 2023-03-29 03:53:40,431 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1680062140431 2023-03-29 03:53:40,431 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 93 msec 2023-03-29 03:53:40,454 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,43505,1680062017306-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-03-29 03:53:40,455 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,43505,1680062017306-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-03-29 03:53:40,455 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,43505,1680062017306-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-03-29 03:53:40,456 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase4:43505, period=300000, unit=MILLISECONDS is enabled. 2023-03-29 03:53:40,457 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-03-29 03:53:40,463 DEBUG [master/jenkins-hbase4:0.Chore.1] janitor.CatalogJanitor(175): 2023-03-29 03:53:40,473 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-03-29 03:53:40,474 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2148): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-03-29 03:53:40,484 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-03-29 03:53:40,487 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-03-29 03:53:40,490 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-03-29 03:53:40,513 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/.tmp/data/hbase/namespace/3fa241e8c8920ab38706390b2493ecb8 2023-03-29 03:53:40,515 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/.tmp/data/hbase/namespace/3fa241e8c8920ab38706390b2493ecb8 empty. 2023-03-29 03:53:40,516 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/.tmp/data/hbase/namespace/3fa241e8c8920ab38706390b2493ecb8 2023-03-29 03:53:40,517 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-03-29 03:53:40,939 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-03-29 03:53:40,942 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7675): creating {ENCODED => 3fa241e8c8920ab38706390b2493ecb8, NAME => 'hbase:namespace,,1680062020474.3fa241e8c8920ab38706390b2493ecb8.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/.tmp 2023-03-29 03:53:41,354 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1680062020474.3fa241e8c8920ab38706390b2493ecb8.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-29 03:53:41,354 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1604): Closing 3fa241e8c8920ab38706390b2493ecb8, disabling compactions & flushes 2023-03-29 03:53:41,354 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1680062020474.3fa241e8c8920ab38706390b2493ecb8. 2023-03-29 03:53:41,354 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1680062020474.3fa241e8c8920ab38706390b2493ecb8. 2023-03-29 03:53:41,354 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1680062020474.3fa241e8c8920ab38706390b2493ecb8. after waiting 0 ms 2023-03-29 03:53:41,354 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1680062020474.3fa241e8c8920ab38706390b2493ecb8. 2023-03-29 03:53:41,354 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1838): Closed hbase:namespace,,1680062020474.3fa241e8c8920ab38706390b2493ecb8. 2023-03-29 03:53:41,354 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1558): Region close journal for 3fa241e8c8920ab38706390b2493ecb8: 2023-03-29 03:53:41,360 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-03-29 03:53:41,379 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1680062020474.3fa241e8c8920ab38706390b2493ecb8.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1680062021364"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680062021364"}]},"ts":"1680062021364"} 2023-03-29 03:53:41,409 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-03-29 03:53:41,411 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-03-29 03:53:41,417 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680062021412"}]},"ts":"1680062021412"} 2023-03-29 03:53:41,422 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-03-29 03:53:41,431 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=3fa241e8c8920ab38706390b2493ecb8, ASSIGN}] 2023-03-29 03:53:41,435 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=3fa241e8c8920ab38706390b2493ecb8, ASSIGN 2023-03-29 03:53:41,437 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=3fa241e8c8920ab38706390b2493ecb8, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,35599,1680062018534; forceNewPlan=false, retain=false 2023-03-29 03:53:41,588 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=3fa241e8c8920ab38706390b2493ecb8, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:41,588 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1680062020474.3fa241e8c8920ab38706390b2493ecb8.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1680062021588"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680062021588"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680062021588"}]},"ts":"1680062021588"} 2023-03-29 03:53:41,593 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure 3fa241e8c8920ab38706390b2493ecb8, server=jenkins-hbase4.apache.org,35599,1680062018534}] 2023-03-29 03:53:41,752 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:namespace,,1680062020474.3fa241e8c8920ab38706390b2493ecb8. 2023-03-29 03:53:41,753 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 3fa241e8c8920ab38706390b2493ecb8, NAME => 'hbase:namespace,,1680062020474.3fa241e8c8920ab38706390b2493ecb8.', STARTKEY => '', ENDKEY => ''} 2023-03-29 03:53:41,754 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace 3fa241e8c8920ab38706390b2493ecb8 2023-03-29 03:53:41,754 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1680062020474.3fa241e8c8920ab38706390b2493ecb8.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-29 03:53:41,754 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 3fa241e8c8920ab38706390b2493ecb8 2023-03-29 03:53:41,754 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 3fa241e8c8920ab38706390b2493ecb8 2023-03-29 03:53:41,757 INFO [StoreOpener-3fa241e8c8920ab38706390b2493ecb8-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 3fa241e8c8920ab38706390b2493ecb8 2023-03-29 03:53:41,759 DEBUG [StoreOpener-3fa241e8c8920ab38706390b2493ecb8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/hbase/namespace/3fa241e8c8920ab38706390b2493ecb8/info 2023-03-29 03:53:41,759 DEBUG [StoreOpener-3fa241e8c8920ab38706390b2493ecb8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/hbase/namespace/3fa241e8c8920ab38706390b2493ecb8/info 2023-03-29 03:53:41,759 INFO [StoreOpener-3fa241e8c8920ab38706390b2493ecb8-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 3fa241e8c8920ab38706390b2493ecb8 columnFamilyName info 2023-03-29 03:53:41,760 INFO [StoreOpener-3fa241e8c8920ab38706390b2493ecb8-1] regionserver.HStore(310): Store=3fa241e8c8920ab38706390b2493ecb8/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-29 03:53:41,762 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/hbase/namespace/3fa241e8c8920ab38706390b2493ecb8 2023-03-29 03:53:41,763 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/hbase/namespace/3fa241e8c8920ab38706390b2493ecb8 2023-03-29 03:53:41,767 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 3fa241e8c8920ab38706390b2493ecb8 2023-03-29 03:53:41,771 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/hbase/namespace/3fa241e8c8920ab38706390b2493ecb8/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-29 03:53:41,771 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 3fa241e8c8920ab38706390b2493ecb8; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66218380, jitterRate=-0.013269245624542236}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-03-29 03:53:41,772 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 3fa241e8c8920ab38706390b2493ecb8: 2023-03-29 03:53:41,773 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:namespace,,1680062020474.3fa241e8c8920ab38706390b2493ecb8., pid=6, masterSystemTime=1680062021747 2023-03-29 03:53:41,777 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:namespace,,1680062020474.3fa241e8c8920ab38706390b2493ecb8. 2023-03-29 03:53:41,777 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:namespace,,1680062020474.3fa241e8c8920ab38706390b2493ecb8. 2023-03-29 03:53:41,778 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=3fa241e8c8920ab38706390b2493ecb8, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:41,779 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1680062020474.3fa241e8c8920ab38706390b2493ecb8.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1680062021778"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1680062021778"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680062021778"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680062021778"}]},"ts":"1680062021778"} 2023-03-29 03:53:41,787 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-03-29 03:53:41,787 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure 3fa241e8c8920ab38706390b2493ecb8, server=jenkins-hbase4.apache.org,35599,1680062018534 in 191 msec 2023-03-29 03:53:41,790 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-03-29 03:53:41,791 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=3fa241e8c8920ab38706390b2493ecb8, ASSIGN in 356 msec 2023-03-29 03:53:41,792 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-03-29 03:53:41,792 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680062021792"}]},"ts":"1680062021792"} 2023-03-29 03:53:41,795 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-03-29 03:53:41,799 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-03-29 03:53:41,801 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 1.3240 sec 2023-03-29 03:53:41,888 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-03-29 03:53:41,894 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-03-29 03:53:41,894 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-29 03:53:41,928 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-03-29 03:53:41,944 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-03-29 03:53:41,953 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 29 msec 2023-03-29 03:53:41,962 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-03-29 03:53:41,975 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-03-29 03:53:41,980 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 17 msec 2023-03-29 03:53:41,988 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-03-29 03:53:41,991 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-03-29 03:53:41,991 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 3.382sec 2023-03-29 03:53:41,993 INFO [master/jenkins-hbase4:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-03-29 03:53:41,995 INFO [master/jenkins-hbase4:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-03-29 03:53:41,995 INFO [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-03-29 03:53:41,996 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,43505,1680062017306-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-03-29 03:53:41,996 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,43505,1680062017306-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-03-29 03:53:42,007 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-03-29 03:53:42,079 DEBUG [Listener at localhost/35027] zookeeper.ReadOnlyZKClient(139): Connect 0x3ddd91d3 to 127.0.0.1:52103 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-29 03:53:42,084 DEBUG [Listener at localhost/35027] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@51f71277, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-29 03:53:42,098 DEBUG [hconnection-0x286751c0-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-29 03:53:42,109 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:45082, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-29 03:53:42,121 INFO [Listener at localhost/35027] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase4.apache.org,43505,1680062017306 2023-03-29 03:53:42,145 INFO [Listener at localhost/35027] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMixedAtomicity Thread=208, OpenFileDescriptor=470, MaxFileDescriptor=60000, SystemLoadAverage=412, ProcessCount=170, AvailableMemoryMB=7714 2023-03-29 03:53:42,163 DEBUG [Listener at localhost/35027] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-03-29 03:53:42,167 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47730, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-03-29 03:53:42,178 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-29 03:53:42,183 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-29 03:53:42,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-03-29 03:53:42,187 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-03-29 03:53:42,189 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-03-29 03:53:42,191 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 9 2023-03-29 03:53:42,192 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/.tmp/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7 2023-03-29 03:53:42,193 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/.tmp/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7 empty. 2023-03-29 03:53:42,195 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/.tmp/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7 2023-03-29 03:53:42,195 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-29 03:53:42,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-03-29 03:53:42,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-03-29 03:53:42,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-03-29 03:53:42,613 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-29 03:53:42,616 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 52c064cf4695189b45c25208ecad32f7, NAME => 'TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/.tmp 2023-03-29 03:53:42,627 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-29 03:53:42,627 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 52c064cf4695189b45c25208ecad32f7, disabling compactions & flushes 2023-03-29 03:53:42,627 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:42,627 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:42,627 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. after waiting 0 ms 2023-03-29 03:53:42,628 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:42,628 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:42,628 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 52c064cf4695189b45c25208ecad32f7: 2023-03-29 03:53:42,632 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-03-29 03:53:42,634 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680062022633"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680062022633"}]},"ts":"1680062022633"} 2023-03-29 03:53:42,637 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-03-29 03:53:42,638 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-03-29 03:53:42,638 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680062022638"}]},"ts":"1680062022638"} 2023-03-29 03:53:42,640 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-03-29 03:53:42,644 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=52c064cf4695189b45c25208ecad32f7, ASSIGN}] 2023-03-29 03:53:42,646 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=52c064cf4695189b45c25208ecad32f7, ASSIGN 2023-03-29 03:53:42,648 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=52c064cf4695189b45c25208ecad32f7, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,35599,1680062018534; forceNewPlan=false, retain=false 2023-03-29 03:53:42,799 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=52c064cf4695189b45c25208ecad32f7, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:42,799 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680062022799"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680062022799"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680062022799"}]},"ts":"1680062022799"} 2023-03-29 03:53:42,803 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=11, ppid=10, state=RUNNABLE; OpenRegionProcedure 52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534}] 2023-03-29 03:53:42,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-03-29 03:53:42,963 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:42,963 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 52c064cf4695189b45c25208ecad32f7, NAME => 'TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7.', STARTKEY => '', ENDKEY => ''} 2023-03-29 03:53:42,964 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 52c064cf4695189b45c25208ecad32f7 2023-03-29 03:53:42,964 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-29 03:53:42,964 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 52c064cf4695189b45c25208ecad32f7 2023-03-29 03:53:42,964 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 52c064cf4695189b45c25208ecad32f7 2023-03-29 03:53:42,969 INFO [StoreOpener-52c064cf4695189b45c25208ecad32f7-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 52c064cf4695189b45c25208ecad32f7 2023-03-29 03:53:42,972 DEBUG [StoreOpener-52c064cf4695189b45c25208ecad32f7-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A 2023-03-29 03:53:42,972 DEBUG [StoreOpener-52c064cf4695189b45c25208ecad32f7-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A 2023-03-29 03:53:42,974 INFO [StoreOpener-52c064cf4695189b45c25208ecad32f7-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-29 03:53:42,975 INFO [StoreOpener-52c064cf4695189b45c25208ecad32f7-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 52c064cf4695189b45c25208ecad32f7 columnFamilyName A 2023-03-29 03:53:42,976 INFO [StoreOpener-52c064cf4695189b45c25208ecad32f7-1] regionserver.HStore(310): Store=52c064cf4695189b45c25208ecad32f7/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-29 03:53:42,976 INFO [StoreOpener-52c064cf4695189b45c25208ecad32f7-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 52c064cf4695189b45c25208ecad32f7 2023-03-29 03:53:42,979 DEBUG [StoreOpener-52c064cf4695189b45c25208ecad32f7-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B 2023-03-29 03:53:42,979 DEBUG [StoreOpener-52c064cf4695189b45c25208ecad32f7-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B 2023-03-29 03:53:42,979 INFO [StoreOpener-52c064cf4695189b45c25208ecad32f7-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-29 03:53:42,980 INFO [StoreOpener-52c064cf4695189b45c25208ecad32f7-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 52c064cf4695189b45c25208ecad32f7 columnFamilyName B 2023-03-29 03:53:42,981 INFO [StoreOpener-52c064cf4695189b45c25208ecad32f7-1] regionserver.HStore(310): Store=52c064cf4695189b45c25208ecad32f7/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-29 03:53:42,981 INFO [StoreOpener-52c064cf4695189b45c25208ecad32f7-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 52c064cf4695189b45c25208ecad32f7 2023-03-29 03:53:42,984 DEBUG [StoreOpener-52c064cf4695189b45c25208ecad32f7-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C 2023-03-29 03:53:42,984 DEBUG [StoreOpener-52c064cf4695189b45c25208ecad32f7-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C 2023-03-29 03:53:42,984 INFO [StoreOpener-52c064cf4695189b45c25208ecad32f7-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-29 03:53:42,985 INFO [StoreOpener-52c064cf4695189b45c25208ecad32f7-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 52c064cf4695189b45c25208ecad32f7 columnFamilyName C 2023-03-29 03:53:42,986 INFO [StoreOpener-52c064cf4695189b45c25208ecad32f7-1] regionserver.HStore(310): Store=52c064cf4695189b45c25208ecad32f7/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-29 03:53:42,987 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:42,989 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7 2023-03-29 03:53:42,990 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7 2023-03-29 03:53:42,994 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-29 03:53:42,998 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 52c064cf4695189b45c25208ecad32f7 2023-03-29 03:53:43,003 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-29 03:53:43,004 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 52c064cf4695189b45c25208ecad32f7; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61952529, jitterRate=-0.07683537900447845}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-29 03:53:43,005 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 52c064cf4695189b45c25208ecad32f7: 2023-03-29 03:53:43,006 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7., pid=11, masterSystemTime=1680062022956 2023-03-29 03:53:43,010 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:43,010 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:43,011 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=52c064cf4695189b45c25208ecad32f7, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:43,011 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680062023011"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1680062023011"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680062023011"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680062023011"}]},"ts":"1680062023011"} 2023-03-29 03:53:43,019 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=11, resume processing ppid=10 2023-03-29 03:53:43,019 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=10, state=SUCCESS; OpenRegionProcedure 52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 in 212 msec 2023-03-29 03:53:43,026 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=10, resume processing ppid=9 2023-03-29 03:53:43,026 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=52c064cf4695189b45c25208ecad32f7, ASSIGN in 375 msec 2023-03-29 03:53:43,027 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-03-29 03:53:43,028 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680062023028"}]},"ts":"1680062023028"} 2023-03-29 03:53:43,031 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-03-29 03:53:43,036 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-03-29 03:53:43,039 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 853 msec 2023-03-29 03:53:43,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-03-29 03:53:43,313 INFO [Listener at localhost/35027] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 9 completed 2023-03-29 03:53:43,318 DEBUG [Listener at localhost/35027] zookeeper.ReadOnlyZKClient(139): Connect 0x2678d495 to 127.0.0.1:52103 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-29 03:53:43,323 DEBUG [Listener at localhost/35027] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@8c3f619, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-29 03:53:43,326 DEBUG [Listener at localhost/35027] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-29 03:53:43,329 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:45094, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-29 03:53:43,332 DEBUG [Listener at localhost/35027] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-03-29 03:53:43,335 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47744, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-03-29 03:53:43,345 DEBUG [Listener at localhost/35027] zookeeper.ReadOnlyZKClient(139): Connect 0x56c8e45e to 127.0.0.1:52103 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-29 03:53:43,350 DEBUG [Listener at localhost/35027] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@e16d41b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-29 03:53:43,350 DEBUG [Listener at localhost/35027] zookeeper.ReadOnlyZKClient(139): Connect 0x5e91d243 to 127.0.0.1:52103 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-29 03:53:43,355 DEBUG [Listener at localhost/35027] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@77648431, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-29 03:53:43,356 DEBUG [Listener at localhost/35027] zookeeper.ReadOnlyZKClient(139): Connect 0x6038a026 to 127.0.0.1:52103 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-29 03:53:43,360 DEBUG [Listener at localhost/35027] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7e728705, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-29 03:53:43,360 DEBUG [Listener at localhost/35027] zookeeper.ReadOnlyZKClient(139): Connect 0x46192d17 to 127.0.0.1:52103 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-29 03:53:43,364 DEBUG [Listener at localhost/35027] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@9a575b8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-29 03:53:43,364 DEBUG [Listener at localhost/35027] zookeeper.ReadOnlyZKClient(139): Connect 0x0dbb9ac1 to 127.0.0.1:52103 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-29 03:53:43,368 DEBUG [Listener at localhost/35027] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@42927198, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-29 03:53:43,369 DEBUG [Listener at localhost/35027] zookeeper.ReadOnlyZKClient(139): Connect 0x6fef7a21 to 127.0.0.1:52103 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-29 03:53:43,375 DEBUG [Listener at localhost/35027] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4fb2b82, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-29 03:53:43,375 DEBUG [Listener at localhost/35027] zookeeper.ReadOnlyZKClient(139): Connect 0x2879e2e1 to 127.0.0.1:52103 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-29 03:53:43,395 DEBUG [Listener at localhost/35027] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3cd3abbd, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-29 03:53:43,396 DEBUG [Listener at localhost/35027] zookeeper.ReadOnlyZKClient(139): Connect 0x6aad2e18 to 127.0.0.1:52103 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-29 03:53:43,400 DEBUG [Listener at localhost/35027] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4b02eeb3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-29 03:53:43,400 DEBUG [Listener at localhost/35027] zookeeper.ReadOnlyZKClient(139): Connect 0x4bd6fc59 to 127.0.0.1:52103 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-29 03:53:43,404 DEBUG [Listener at localhost/35027] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4808c6d5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-29 03:53:43,410 DEBUG [hconnection-0x21233b14-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-29 03:53:43,411 DEBUG [hconnection-0x43a25080-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-29 03:53:43,413 DEBUG [hconnection-0x20746307-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-29 03:53:43,413 DEBUG [hconnection-0x3df3bc44-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-29 03:53:43,413 DEBUG [hconnection-0x1a5dd2ff-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-29 03:53:43,415 DEBUG [hconnection-0x2a12d69-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-29 03:53:43,418 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48198, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-29 03:53:43,424 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48182, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-29 03:53:43,425 DEBUG [hconnection-0xce38a73-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-29 03:53:43,427 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48208, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-29 03:53:43,428 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48236, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-29 03:53:43,428 DEBUG [hconnection-0x27e010-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-29 03:53:43,429 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48220, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-29 03:53:43,433 DEBUG [hconnection-0x2c5c1135-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-29 03:53:43,436 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48250, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-29 03:53:43,439 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48240, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-29 03:53:43,439 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48252, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-29 03:53:43,447 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:53:43,449 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48262, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-29 03:53:43,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:53:43,467 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:53:43,467 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:53:43,468 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:53:43,468 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:53:43,469 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:43,469 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:43,472 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:53:43,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:53:43,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:53:43,472 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:43,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:43,473 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:43,473 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:43,474 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:53:43,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:53:43,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:43,474 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:53:43,477 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:53:43,481 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:53:43,481 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:53:43,484 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:53:43,485 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:53:43,485 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:53:43,486 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:43,487 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. started... 2023-03-29 03:53:43,490 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 52c064cf4695189b45c25208ecad32f7 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-03-29 03:53:43,503 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 52c064cf4695189b45c25208ecad32f7, store=A 2023-03-29 03:53:43,504 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:53:43,504 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 52c064cf4695189b45c25208ecad32f7, store=B 2023-03-29 03:53:43,505 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:53:43,505 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 52c064cf4695189b45c25208ecad32f7, store=C 2023-03-29 03:53:43,505 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:53:43,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on 52c064cf4695189b45c25208ecad32f7 2023-03-29 03:53:43,517 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. as already flushing 2023-03-29 03:53:43,648 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680062023573, max timestamp=1680062023608Num uniques 859; 2023-03-29 03:53:43,649 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-03-29 03:53:43,656 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680062023573, max timestamp=1680062023609Num uniques 859; 2023-03-29 03:53:43,656 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-03-29 03:53:43,656 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680062023573, max timestamp=1680062023612Num uniques 859; 2023-03-29 03:53:43,686 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-03-29 03:53:43,695 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=8 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/A/8b45c8fe9a884504ade2b78fafe29d39 2023-03-29 03:53:43,736 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680062023705, max timestamp=1680062023726Num uniques 1009; 2023-03-29 03:53:43,736 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46213, new segment data size=6870, suffix heap size=132904,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=150 2023-03-29 03:53:43,740 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680062023705, max timestamp=1680062023727Num uniques 1009; 2023-03-29 03:53:43,741 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46206, new segment data size=6870, suffix heap size=132904,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=150 2023-03-29 03:53:43,743 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680062023705, max timestamp=1680062023727Num uniques 1009; 2023-03-29 03:53:43,743 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46213, new segment data size=6870, suffix heap size=132904,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=150 2023-03-29 03:53:43,805 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=8 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/B/ed42bd0be8ee4e5ab80efac882a336c7 2023-03-29 03:53:43,807 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8092, totalHeapSize=18648, min timestamp=1680062023767, max timestamp=1680062023779Num uniques 1009; 2023-03-29 03:53:43,808 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46211, new segment data size=8092, suffix heap size=132904,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=177 2023-03-29 03:53:43,809 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1680062023767, max timestamp=1680062023779Num uniques 1009; 2023-03-29 03:53:43,809 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=8102, suffix heap size=132904,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=177 2023-03-29 03:53:43,810 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8092, totalHeapSize=18648, min timestamp=1680062023767, max timestamp=1680062023779Num uniques 1009; 2023-03-29 03:53:43,810 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46204, new segment data size=8092, suffix heap size=132904,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=177 2023-03-29 03:53:43,851 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1680062023835, max timestamp=1680062023844Num uniques 1036; 2023-03-29 03:53:43,851 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-03-29 03:53:43,852 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1680062023835, max timestamp=1680062023844Num uniques 1036; 2023-03-29 03:53:43,852 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-03-29 03:53:43,857 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680062023835, max timestamp=1680062023842Num uniques 1036; 2023-03-29 03:53:43,857 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=6870, suffix heap size=135712,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=150 2023-03-29 03:53:43,897 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1680062023856, max timestamp=1680062023892Num uniques 1009; 2023-03-29 03:53:43,897 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=11220, suffix heap size=132904,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=245 2023-03-29 03:53:43,899 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680062023856, max timestamp=1680062023892Num uniques 1045; 2023-03-29 03:53:43,899 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=6870, suffix heap size=136648,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=150 2023-03-29 03:53:43,902 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680062023856, max timestamp=1680062023892Num uniques 1045; 2023-03-29 03:53:43,902 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=6870, suffix heap size=136648,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=150 2023-03-29 03:53:43,931 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9334, totalHeapSize=21456, min timestamp=1680062023920, max timestamp=1680062023926Num uniques 1104; 2023-03-29 03:53:43,931 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=50554, new segment data size=9334, suffix heap size=142784,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=204 2023-03-29 03:53:43,931 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680062023920, max timestamp=1680062023926Num uniques 1009; 2023-03-29 03:53:43,931 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9336, totalHeapSize=21456, min timestamp=1680062023920, max timestamp=1680062023926Num uniques 1009; 2023-03-29 03:53:43,931 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46204, new segment data size=6870, suffix heap size=132904,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=150 2023-03-29 03:53:43,932 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46206, new segment data size=9336, suffix heap size=132904,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=204 2023-03-29 03:53:43,970 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1680062023954, max timestamp=1680062023963Num uniques 1063; 2023-03-29 03:53:43,970 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=7458, suffix heap size=138520,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=163 2023-03-29 03:53:43,970 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7456, totalHeapSize=17192, min timestamp=1680062023954, max timestamp=1680062023965Num uniques 1063; 2023-03-29 03:53:43,970 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=48676, new segment data size=7456, suffix heap size=138520,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=163 2023-03-29 03:53:43,970 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1680062023954, max timestamp=1680062023964Num uniques 1009; 2023-03-29 03:53:43,971 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=7458, suffix heap size=132904,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=163 2023-03-29 03:53:43,999 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1680062023985, max timestamp=1680062023994Num uniques 1022; 2023-03-29 03:53:43,999 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=10162, suffix heap size=134256,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=222 2023-03-29 03:53:43,999 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1680062023985, max timestamp=1680062023994Num uniques 1022; 2023-03-29 03:53:43,999 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1680062023985, max timestamp=1680062023994Num uniques 1022; 2023-03-29 03:53:43,999 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=10162, suffix heap size=134256,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=222 2023-03-29 03:53:43,999 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=10162, suffix heap size=134256,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=222 2023-03-29 03:53:44,032 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=381, cellSize=17446, totalHeapSize=39864, min timestamp=1680062024003, max timestamp=1680062024028Num uniques 1081; 2023-03-29 03:53:44,032 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=17446, suffix heap size=140392,new segment heap size=39864 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=381 2023-03-29 03:53:44,032 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=381, cellSize=17446, totalHeapSize=39864, min timestamp=1680062024003, max timestamp=1680062024028Num uniques 1081; 2023-03-29 03:53:44,033 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=17446, suffix heap size=140392,new segment heap size=39864 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=381 2023-03-29 03:53:44,035 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1680062024020, max timestamp=1680062024028Num uniques 1081; 2023-03-29 03:53:44,035 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=8286, suffix heap size=140392,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=181 2023-03-29 03:53:44,076 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:44,077 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:44,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48262 deadline: 1680062084070, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:44,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48182 deadline: 1680062084072, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:44,078 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1680062024054, max timestamp=1680062024072Num uniques 1040; 2023-03-29 03:53:44,078 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1680062024054, max timestamp=1680062024072Num uniques 1240; 2023-03-29 03:53:44,078 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=8700, suffix heap size=136128,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=190 2023-03-29 03:53:44,078 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=56790, new segment data size=8700, suffix heap size=156928,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1240, new segment cells count=190 2023-03-29 03:53:44,078 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1680062024054, max timestamp=1680062024072Num uniques 1240; 2023-03-29 03:53:44,079 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=56790, new segment data size=8700, suffix heap size=156928,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1240, new segment cells count=190 2023-03-29 03:53:44,144 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1680062024123, max timestamp=1680062024139Num uniques 1049; 2023-03-29 03:53:44,144 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=48044, new segment data size=9114, suffix heap size=137064,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=199 2023-03-29 03:53:44,146 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1680062024123, max timestamp=1680062024142Num uniques 1049; 2023-03-29 03:53:44,146 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=48044, new segment data size=9114, suffix heap size=137064,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=199 2023-03-29 03:53:44,146 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1680062024123, max timestamp=1680062024142Num uniques 1049; 2023-03-29 03:53:44,146 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48044, new segment data size=9114, suffix heap size=137064,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=199 2023-03-29 03:53:44,184 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=258, cellSize=11810, totalHeapSize=27072, min timestamp=1680062024162, max timestamp=1680062024181Num uniques 1058; 2023-03-29 03:53:44,185 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=11810, suffix heap size=138000,new segment heap size=27072 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=258 2023-03-29 03:53:44,187 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=258, cellSize=11810, totalHeapSize=27072, min timestamp=1680062024162, max timestamp=1680062024181Num uniques 1058; 2023-03-29 03:53:44,187 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=11810, suffix heap size=138000,new segment heap size=27072 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=258 2023-03-29 03:53:44,187 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=258, cellSize=11810, totalHeapSize=27072, min timestamp=1680062024162, max timestamp=1680062024181Num uniques 1058; 2023-03-29 03:53:44,187 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=11810, suffix heap size=138000,new segment heap size=27072 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=258 2023-03-29 03:53:44,232 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=167, cellSize=7642, totalHeapSize=17608, min timestamp=1680062024219, max timestamp=1680062024224Num uniques 1117; 2023-03-29 03:53:44,232 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=51152, new segment data size=7642, suffix heap size=144136,new segment heap size=17608 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1117, new segment cells count=167 2023-03-29 03:53:44,232 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=167, cellSize=7642, totalHeapSize=17608, min timestamp=1680062024219, max timestamp=1680062024224Num uniques 1117; 2023-03-29 03:53:44,232 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=167, cellSize=7642, totalHeapSize=17608, min timestamp=1680062024219, max timestamp=1680062024224Num uniques 1117; 2023-03-29 03:53:44,233 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=51152, new segment data size=7642, suffix heap size=144136,new segment heap size=17608 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1117, new segment cells count=167 2023-03-29 03:53:44,233 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=51152, new segment data size=7642, suffix heap size=144136,new segment heap size=17608 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1117, new segment cells count=167 2023-03-29 03:53:44,282 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=8 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/C/1cc6b52678be4b239ffa8879a819583f 2023-03-29 03:53:44,294 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/A/8b45c8fe9a884504ade2b78fafe29d39 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/8b45c8fe9a884504ade2b78fafe29d39 2023-03-29 03:53:44,317 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/8b45c8fe9a884504ade2b78fafe29d39, entries=100, sequenceid=8, filesize=9.3 K 2023-03-29 03:53:44,326 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=376, cellSize=17216, totalHeapSize=39344, min timestamp=1680062024242, max timestamp=1680062024317Num uniques 1026; 2023-03-29 03:53:44,329 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46986, new segment data size=17216, suffix heap size=134672,new segment heap size=39344 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1026, new segment cells count=376 2023-03-29 03:53:44,331 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=376, cellSize=17216, totalHeapSize=39344, min timestamp=1680062024242, max timestamp=1680062024317Num uniques 1026; 2023-03-29 03:53:44,331 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46986, new segment data size=17216, suffix heap size=134672,new segment heap size=39344 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1026, new segment cells count=376 2023-03-29 03:53:44,332 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=376, cellSize=17216, totalHeapSize=39344, min timestamp=1680062024242, max timestamp=1680062024317Num uniques 1026; 2023-03-29 03:53:44,333 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46986, new segment data size=17216, suffix heap size=134672,new segment heap size=39344 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1026, new segment cells count=376 2023-03-29 03:53:44,335 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/B/ed42bd0be8ee4e5ab80efac882a336c7 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/ed42bd0be8ee4e5ab80efac882a336c7 2023-03-29 03:53:44,371 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/ed42bd0be8ee4e5ab80efac882a336c7, entries=100, sequenceid=8, filesize=9.3 K 2023-03-29 03:53:44,378 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/C/1cc6b52678be4b239ffa8879a819583f as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/1cc6b52678be4b239ffa8879a819583f 2023-03-29 03:53:44,419 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/1cc6b52678be4b239ffa8879a819583f, entries=100, sequenceid=8, filesize=9.3 K 2023-03-29 03:53:44,434 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=93.93 KB/96180 for 52c064cf4695189b45c25208ecad32f7 in 944ms, sequenceid=8, compaction requested=false 2023-03-29 03:53:44,436 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-03-29 03:53:44,439 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 52c064cf4695189b45c25208ecad32f7: 2023-03-29 03:53:44,439 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:44,440 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:53:44,440 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:53:44,440 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:44,441 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:53:44,441 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:53:44,445 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:44,445 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:44,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:44,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:44,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:44,446 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:44,446 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:53:44,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:44,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:44,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:44,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:44,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:44,449 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:53:44,449 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2332f169[Count = 0] remaining members to acquire global barrier 2023-03-29 03:53:44,449 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:53:44,449 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:44,453 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:44,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:44,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:44,453 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:53:44,453 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:53:44,453 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:53:44,453 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:44,453 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:53:44,457 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:44,457 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:53:44,457 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:53:44,457 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:53:44,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:44,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:44,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:44,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:44,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:44,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:44,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:44,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:44,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:44,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:44,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:53:44,463 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:53:44,463 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:53:44,463 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:53:44,463 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:53:44,463 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:53:44,463 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:53:44,466 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:44,466 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:44,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:44,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:44,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:44,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:44,466 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:53:44,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:44,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:44,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:53:44,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:53:44,468 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:44,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:44,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:44,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:44,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:44,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:44,471 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:44,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:44,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:44,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:44,481 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:44,481 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:53:44,481 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:44,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:53:44,482 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:44,482 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:44,481 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:53:44,482 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:44,482 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:53:44,482 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:53:44,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:53:44,483 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:53:44,482 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:44,483 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:44,485 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:53:44,485 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:53:44,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:53:44,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:53:44,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on 52c064cf4695189b45c25208ecad32f7 2023-03-29 03:53:44,496 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 52c064cf4695189b45c25208ecad32f7 3/3 column families, dataSize=127.47 KB heapSize=344.72 KB 2023-03-29 03:53:44,497 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 52c064cf4695189b45c25208ecad32f7, store=A 2023-03-29 03:53:44,497 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-29 03:53:44,497 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 52c064cf4695189b45c25208ecad32f7, store=B 2023-03-29 03:53:44,497 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-29 03:53:44,497 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 52c064cf4695189b45c25208ecad32f7, store=C 2023-03-29 03:53:44,498 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-29 03:53:44,550 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:44,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48252 deadline: 1680062084536, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:44,553 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:44,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48198 deadline: 1680062084543, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:44,560 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=263 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/A/3310d1d5d2af4927834788258e0ea18a 2023-03-29 03:53:44,561 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:44,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48182 deadline: 1680062084548, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:44,562 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:44,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48262 deadline: 1680062084550, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:44,574 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:44,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48240 deadline: 1680062084554, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:44,585 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:53:44,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:53:44,622 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:53:44,631 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=263 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/B/f5b1d536c98f4340b12e0a35f5153201 2023-03-29 03:53:44,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:53:44,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:53:44,633 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:53:44,633 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:53:44,634 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:53:44,634 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:53:44,635 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:44,635 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:44,638 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:53:44,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:53:44,638 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:44,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:53:44,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:44,640 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:44,640 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:53:44,640 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:44,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:53:44,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:44,641 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:53:44,641 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:53:44,641 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:53:44,641 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:53:44,642 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:53:44,645 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:53:44,651 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:53:44,651 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:53:44,656 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:44,657 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. started... 2023-03-29 03:53:44,657 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. as already flushing 2023-03-29 03:53:44,657 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:44,658 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:53:44,658 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:53:44,658 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:44,658 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:53:44,658 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:53:44,662 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:44,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:44,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:44,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:44,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:44,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:44,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:44,664 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:44,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:44,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:44,668 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:53:44,668 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@246023e0[Count = 0] remaining members to acquire global barrier 2023-03-29 03:53:44,669 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:53:44,669 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:44,670 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:44,670 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:44,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48252 deadline: 1680062084654, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:44,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48198 deadline: 1680062084657, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:44,671 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(162): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:44,671 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:44,671 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:53:44,671 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:53:44,671 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:53:44,671 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:53:44,671 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:44,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48182 deadline: 1680062084665, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:44,672 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:44,672 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:44,672 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:53:44,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48262 deadline: 1680062084666, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:44,678 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:53:44,678 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:53:44,678 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:44,678 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:53:44,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:44,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:44,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:44,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:44,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:44,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:44,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:44,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:44,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:44,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:44,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:53:44,684 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:53:44,684 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:53:44,685 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:53:44,685 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:53:44,685 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:53:44,685 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:53:44,691 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:44,691 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:53:44,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:44,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:44,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:53:44,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:53:44,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:44,697 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:44,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:44,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:44,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:44,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:44,701 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:44,702 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:44,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48240 deadline: 1680062084681, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:44,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:44,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:44,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:44,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:44,705 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:44,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:44,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:44,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:44,714 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:53:44,715 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:44,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:53:44,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:53:44,715 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:53:44,715 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:53:44,715 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:53:44,715 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:53:44,716 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:53:44,715 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:44,716 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:44,716 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:44,716 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:44,716 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:44,716 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:53:44,716 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:53:44,716 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:44,717 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:44,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:53:44,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:53:44,817 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:53:44,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:53:44,824 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:53:44,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:53:44,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:53:44,836 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:53:44,836 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:53:44,837 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:53:44,837 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:53:44,839 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:44,839 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:44,840 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:44,841 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:53:44,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:53:44,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:53:44,841 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:44,841 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:53:44,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:44,842 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:44,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:53:44,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:44,843 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:53:44,848 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:53:44,848 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:53:44,849 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:53:44,849 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:53:44,851 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:53:44,852 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:53:44,852 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:53:44,852 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:44,852 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. started... 2023-03-29 03:53:44,852 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. as already flushing 2023-03-29 03:53:44,852 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:44,856 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:53:44,856 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:53:44,856 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:44,856 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:53:44,856 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:53:44,859 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:44,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:44,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:44,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:44,860 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:44,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:44,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:44,861 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:44,861 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:53:44,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:44,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:44,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:44,863 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:53:44,863 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e6d2d98[Count = 0] remaining members to acquire global barrier 2023-03-29 03:53:44,863 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:53:44,863 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:44,866 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:44,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:44,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:44,867 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:53:44,867 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:53:44,867 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:53:44,867 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:44,867 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:53:44,871 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:53:44,871 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:53:44,871 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:53:44,872 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:44,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:44,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:44,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:44,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:44,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:44,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:44,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:44,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:44,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:44,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:44,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:53:44,881 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:53:44,881 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:53:44,881 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:53:44,881 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:53:44,882 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:53:44,882 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:53:44,884 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:44,884 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:53:44,884 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:44,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:44,885 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:44,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:53:44,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:53:44,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:44,886 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:44,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:44,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:44,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:44,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:44,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:44,890 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:44,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:44,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48198 deadline: 1680062084876, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:44,890 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:44,890 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:44,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48262 deadline: 1680062084877, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:44,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:44,891 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:44,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48252 deadline: 1680062084877, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:44,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:44,892 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:44,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48182 deadline: 1680062084877, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:44,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:44,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:44,902 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:53:44,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:53:44,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:53:44,902 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:53:44,902 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:44,902 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:53:44,902 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:53:44,904 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:53:44,903 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:44,905 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:44,905 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:44,905 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:44,905 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:44,905 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:44,905 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:53:44,905 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:53:44,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:53:44,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:53:44,905 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:53:44,908 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:44,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48240 deadline: 1680062084907, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:45,005 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:53:45,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:53:45,011 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:53:45,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:53:45,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:53:45,014 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:53:45,014 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:53:45,016 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:53:45,017 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:53:45,017 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:45,017 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:45,020 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:53:45,020 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:45,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:53:45,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:53:45,021 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:45,021 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:53:45,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:45,021 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:45,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:53:45,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:45,022 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:53:45,023 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:53:45,023 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:53:45,024 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:53:45,024 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:53:45,028 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:53:45,029 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:53:45,029 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:53:45,029 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:45,029 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. started... 2023-03-29 03:53:45,029 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. as already flushing 2023-03-29 03:53:45,029 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:45,030 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:53:45,030 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:53:45,030 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:45,030 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:53:45,030 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:53:45,033 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:45,033 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:45,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:45,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:45,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:45,034 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:45,034 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:53:45,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:45,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:45,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:45,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:45,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:45,043 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:53:45,043 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@59f51a5[Count = 0] remaining members to acquire global barrier 2023-03-29 03:53:45,043 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:53:45,043 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:45,046 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:45,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:45,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:45,046 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:53:45,046 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:53:45,046 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:53:45,047 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:45,047 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:53:45,050 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:53:45,050 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:45,050 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:53:45,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:45,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:45,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:45,050 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:53:45,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:45,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:45,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:45,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:45,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:45,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:45,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:45,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:53:45,055 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:53:45,055 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:53:45,055 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:53:45,056 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:53:45,056 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:53:45,056 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:53:45,058 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:45,058 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:45,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:45,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:45,058 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:53:45,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:45,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:45,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:45,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:53:45,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:53:45,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:45,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:45,060 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:45,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:45,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:45,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:45,063 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:45,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:45,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:45,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:45,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:45,078 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:53:45,078 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:53:45,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:53:45,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:53:45,078 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:45,078 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:45,078 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:45,078 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:45,078 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:45,078 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:45,078 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:45,078 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:53:45,079 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:53:45,079 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:53:45,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:53:45,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:53:45,081 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:53:45,084 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:53:45,085 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:53:45,112 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=263 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/C/5fdd7a7a65c6418887a869fa3d363115 2023-03-29 03:53:45,135 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/A/3310d1d5d2af4927834788258e0ea18a as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/3310d1d5d2af4927834788258e0ea18a 2023-03-29 03:53:45,150 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/3310d1d5d2af4927834788258e0ea18a, entries=150, sequenceid=263, filesize=11.9 K 2023-03-29 03:53:45,153 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/B/f5b1d536c98f4340b12e0a35f5153201 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/f5b1d536c98f4340b12e0a35f5153201 2023-03-29 03:53:45,164 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/f5b1d536c98f4340b12e0a35f5153201, entries=150, sequenceid=263, filesize=11.9 K 2023-03-29 03:53:45,173 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/C/5fdd7a7a65c6418887a869fa3d363115 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/5fdd7a7a65c6418887a869fa3d363115 2023-03-29 03:53:45,185 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:53:45,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:53:45,188 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/5fdd7a7a65c6418887a869fa3d363115, entries=150, sequenceid=263, filesize=11.9 K 2023-03-29 03:53:45,191 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~344.67 KB/352944, currentSize=67.09 KB/68700 for 52c064cf4695189b45c25208ecad32f7 in 694ms, sequenceid=263, compaction requested=false 2023-03-29 03:53:45,191 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:53:45,192 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 52c064cf4695189b45c25208ecad32f7: 2023-03-29 03:53:45,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:53:45,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:53:45,200 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:53:45,200 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:53:45,201 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:53:45,201 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:53:45,202 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:45,202 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:45,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on 52c064cf4695189b45c25208ecad32f7 2023-03-29 03:53:45,202 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 52c064cf4695189b45c25208ecad32f7 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-03-29 03:53:45,203 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 52c064cf4695189b45c25208ecad32f7, store=A 2023-03-29 03:53:45,204 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:53:45,204 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 52c064cf4695189b45c25208ecad32f7, store=B 2023-03-29 03:53:45,204 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:53:45,204 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:53:45,204 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 52c064cf4695189b45c25208ecad32f7, store=C 2023-03-29 03:53:45,204 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:53:45,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:53:45,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:53:45,204 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:45,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:45,207 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:45,207 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:53:45,207 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:45,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:53:45,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:45,208 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:53:45,212 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:53:45,213 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:53:45,213 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:53:45,213 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:53:45,219 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:53:45,220 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:53:45,220 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:53:45,220 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:45,220 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. started... 2023-03-29 03:53:45,220 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. as already flushing 2023-03-29 03:53:45,221 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:45,222 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:53:45,222 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:53:45,222 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:45,222 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:53:45,222 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:53:45,224 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:45,224 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:45,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:45,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:45,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:45,225 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:45,225 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:53:45,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:45,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:45,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:45,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:45,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:45,227 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:53:45,227 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@319251c6[Count = 0] remaining members to acquire global barrier 2023-03-29 03:53:45,227 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:53:45,227 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:45,229 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:45,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:45,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:45,230 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:53:45,230 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:53:45,230 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:53:45,230 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:45,230 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:53:45,237 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:53:45,237 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:45,237 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:53:45,237 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:53:45,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:45,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:45,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:45,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:45,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:45,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:45,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:45,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:45,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:45,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:45,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:53:45,243 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:53:45,243 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:53:45,243 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:53:45,243 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:53:45,243 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:53:45,243 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:53:45,249 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:45,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:45,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:45,253 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:45,253 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:53:45,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:45,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:45,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:45,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:45,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:53:45,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:53:45,262 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:45,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:45,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:45,265 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=278 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/A/03d1712dec6b499bb8ad7f3380461caa 2023-03-29 03:53:45,267 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:45,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48252 deadline: 1680062085246, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:45,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:45,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:45,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:45,269 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:45,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:45,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:45,271 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:45,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48240 deadline: 1680062085255, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:45,272 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:45,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48262 deadline: 1680062085256, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:45,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:45,275 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:45,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48198 deadline: 1680062085265, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:45,275 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:45,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48182 deadline: 1680062085267, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:45,280 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:53:45,280 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:45,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:53:45,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:53:45,284 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:53:45,284 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:53:45,284 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:53:45,285 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:53:45,285 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:53:45,280 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:45,280 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:53:45,290 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:45,290 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:45,290 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:45,290 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:45,290 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:45,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:53:45,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:53:45,364 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=278 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/B/f514d196bbc04215b3d0f41b8fcad2db 2023-03-29 03:53:45,371 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:45,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48252 deadline: 1680062085370, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:45,385 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:53:45,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:53:45,388 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:45,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48262 deadline: 1680062085380, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:45,389 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:45,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48182 deadline: 1680062085380, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:45,389 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:45,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48240 deadline: 1680062085380, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:45,390 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:53:45,390 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:45,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48198 deadline: 1680062085380, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:45,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:53:45,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:53:45,402 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:53:45,402 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:53:45,405 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:53:45,405 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:53:45,412 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:45,413 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:45,417 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:45,417 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:53:45,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:53:45,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:53:45,418 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:45,418 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:53:45,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:45,418 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:45,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:53:45,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:45,419 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:53:45,419 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:53:45,419 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:53:45,419 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:53:45,419 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:53:45,420 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:53:45,424 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:53:45,429 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=278 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/C/b817ca9e28eb4d1c87686e612bded898 2023-03-29 03:53:45,429 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:45,429 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:53:45,429 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. started... 2023-03-29 03:53:45,429 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. as already flushing 2023-03-29 03:53:45,429 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:45,430 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:53:45,430 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:53:45,430 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:45,430 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:53:45,430 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:53:45,433 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:45,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:45,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:45,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:45,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:45,435 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:45,436 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:45,436 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:53:45,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:45,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:45,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:45,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:45,438 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:53:45,438 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@14481b37[Count = 0] remaining members to acquire global barrier 2023-03-29 03:53:45,438 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:53:45,438 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:45,440 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:45,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:45,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:45,440 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:53:45,440 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:53:45,441 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:53:45,442 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:45,443 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:53:45,444 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:53:45,444 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:53:45,444 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:53:45,444 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:45,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:45,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:45,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:45,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:45,448 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/A/03d1712dec6b499bb8ad7f3380461caa as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/03d1712dec6b499bb8ad7f3380461caa 2023-03-29 03:53:45,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:45,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:45,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:45,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:45,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:45,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:45,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:53:45,455 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:53:45,455 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:53:45,455 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:53:45,455 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:53:45,455 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:53:45,455 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:53:45,461 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:45,461 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:53:45,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:45,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:45,461 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:45,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:45,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:45,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:45,462 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/03d1712dec6b499bb8ad7f3380461caa, entries=150, sequenceid=278, filesize=11.9 K 2023-03-29 03:53:45,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:53:45,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:53:45,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:45,464 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:45,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:45,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:45,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:45,466 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/B/f514d196bbc04215b3d0f41b8fcad2db as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/f514d196bbc04215b3d0f41b8fcad2db 2023-03-29 03:53:45,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:45,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:45,467 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:45,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:45,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:45,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:45,471 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:53:45,471 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:53:45,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:53:45,471 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:45,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:53:45,471 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:45,471 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:45,471 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:45,472 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:45,472 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:45,472 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:45,472 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:53:45,472 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:53:45,472 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:53:45,472 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:53:45,473 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:53:45,473 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:53:45,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:53:45,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:53:45,478 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/f514d196bbc04215b3d0f41b8fcad2db, entries=150, sequenceid=278, filesize=11.9 K 2023-03-29 03:53:45,483 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/C/b817ca9e28eb4d1c87686e612bded898 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/b817ca9e28eb4d1c87686e612bded898 2023-03-29 03:53:45,506 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/b817ca9e28eb4d1c87686e612bded898, entries=150, sequenceid=278, filesize=11.9 K 2023-03-29 03:53:45,508 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 52c064cf4695189b45c25208ecad32f7 in 305ms, sequenceid=278, compaction requested=true 2023-03-29 03:53:45,509 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 52c064cf4695189b45c25208ecad32f7: 2023-03-29 03:53:45,511 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:53:45,511 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:53:45,512 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:53:45,512 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-29 03:53:45,513 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-29 03:53:45,516 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 33864 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-29 03:53:45,519 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1912): 52c064cf4695189b45c25208ecad32f7/B is initiating minor compaction (all files) 2023-03-29 03:53:45,519 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 52c064cf4695189b45c25208ecad32f7/B in TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:45,519 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/ed42bd0be8ee4e5ab80efac882a336c7, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/f5b1d536c98f4340b12e0a35f5153201, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/f514d196bbc04215b3d0f41b8fcad2db] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp, totalSize=33.1 K 2023-03-29 03:53:45,524 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting ed42bd0be8ee4e5ab80efac882a336c7, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=8, earliestPutTs=1680062023450 2023-03-29 03:53:45,524 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 33864 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-29 03:53:45,530 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1912): 52c064cf4695189b45c25208ecad32f7/A is initiating minor compaction (all files) 2023-03-29 03:53:45,530 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 52c064cf4695189b45c25208ecad32f7/A in TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:45,530 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/8b45c8fe9a884504ade2b78fafe29d39, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/3310d1d5d2af4927834788258e0ea18a, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/03d1712dec6b499bb8ad7f3380461caa] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp, totalSize=33.1 K 2023-03-29 03:53:45,531 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting f5b1d536c98f4340b12e0a35f5153201, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=263, earliestPutTs=1680062024313 2023-03-29 03:53:45,531 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 8b45c8fe9a884504ade2b78fafe29d39, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=8, earliestPutTs=1680062023450 2023-03-29 03:53:45,533 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting f514d196bbc04215b3d0f41b8fcad2db, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=278, earliestPutTs=1680062024522 2023-03-29 03:53:45,536 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 3310d1d5d2af4927834788258e0ea18a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=263, earliestPutTs=1680062024313 2023-03-29 03:53:45,538 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 03d1712dec6b499bb8ad7f3380461caa, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=278, earliestPutTs=1680062024522 2023-03-29 03:53:45,575 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:53:45,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:53:45,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on 52c064cf4695189b45c25208ecad32f7 2023-03-29 03:53:45,582 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 52c064cf4695189b45c25208ecad32f7 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-03-29 03:53:45,582 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 52c064cf4695189b45c25208ecad32f7, store=A 2023-03-29 03:53:45,583 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:53:45,583 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:53:45,583 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 52c064cf4695189b45c25208ecad32f7, store=B 2023-03-29 03:53:45,583 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:53:45,583 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] throttle.PressureAwareThroughputController(145): 52c064cf4695189b45c25208ecad32f7#A#compaction#9 average throughput is 0.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:53:45,583 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 52c064cf4695189b45c25208ecad32f7, store=C 2023-03-29 03:53:45,583 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:53:45,604 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] throttle.PressureAwareThroughputController(145): 52c064cf4695189b45c25208ecad32f7#B#compaction#10 average throughput is 0.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:53:45,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:53:45,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:53:45,605 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:53:45,605 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:53:45,606 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:53:45,606 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:53:45,607 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:45,608 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:45,613 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:53:45,613 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:45,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:53:45,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:53:45,614 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:45,614 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:53:45,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:45,615 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:45,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:53:45,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:45,615 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:53:45,616 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:53:45,616 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:53:45,616 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:53:45,621 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:53:45,625 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:53:45,628 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:53:45,628 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:53:45,629 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:45,629 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. started... 2023-03-29 03:53:45,629 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. as already flushing 2023-03-29 03:53:45,629 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:45,631 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:53:45,631 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:53:45,631 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:45,632 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:53:45,632 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:53:45,637 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:45,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:45,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:45,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:45,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:45,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:45,638 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:45,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:45,639 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:45,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:45,639 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:53:45,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:45,640 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:53:45,640 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@732ef5cb[Count = 0] remaining members to acquire global barrier 2023-03-29 03:53:45,640 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:53:45,640 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:45,642 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:45,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:45,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:45,642 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:53:45,643 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:53:45,643 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:53:45,643 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:45,643 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:53:45,644 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:53:45,644 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:45,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:45,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:45,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:45,644 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:53:45,651 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:53:45,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:45,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:45,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:45,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:45,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:45,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:45,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:45,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:53:45,659 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:53:45,659 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:53:45,659 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:53:45,660 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:53:45,660 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:53:45,660 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:53:45,663 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:45,663 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:53:45,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:45,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:45,664 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:45,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:45,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:45,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:45,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:53:45,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:53:45,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:45,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:45,670 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:45,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:45,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:45,671 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=296 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/A/d4d0ac16fd8445fe94434c01ed615900 2023-03-29 03:53:45,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:45,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:45,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:45,673 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:45,674 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/A/ea13c770aa4b4756b2fb2465c8878fe4 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/ea13c770aa4b4756b2fb2465c8878fe4 2023-03-29 03:53:45,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:45,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:45,675 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:45,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48240 deadline: 1680062085658, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:45,677 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:53:45,677 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:53:45,677 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:45,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:53:45,677 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:45,678 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:53:45,678 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:45,678 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:45,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:53:45,678 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:45,678 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:53:45,681 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:53:45,680 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:45,681 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:45,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:53:45,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:53:45,682 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:53:45,684 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:53:45,686 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:45,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48182 deadline: 1680062085669, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:45,687 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:45,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48252 deadline: 1680062085670, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:45,687 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:45,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48198 deadline: 1680062085671, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:45,688 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:45,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48262 deadline: 1680062085672, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:45,723 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 52c064cf4695189b45c25208ecad32f7/A of 52c064cf4695189b45c25208ecad32f7 into ea13c770aa4b4756b2fb2465c8878fe4(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:53:45,723 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=296 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/B/e9991746cacf48aba8504b1506edde5b 2023-03-29 03:53:45,723 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 52c064cf4695189b45c25208ecad32f7: 2023-03-29 03:53:45,723 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7., storeName=52c064cf4695189b45c25208ecad32f7/A, priority=13, startTime=1680062025511; duration=0sec 2023-03-29 03:53:45,724 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:53:45,724 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-29 03:53:45,729 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 33864 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-29 03:53:45,729 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1912): 52c064cf4695189b45c25208ecad32f7/C is initiating minor compaction (all files) 2023-03-29 03:53:45,729 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 52c064cf4695189b45c25208ecad32f7/C in TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:45,729 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/1cc6b52678be4b239ffa8879a819583f, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/5fdd7a7a65c6418887a869fa3d363115, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/b817ca9e28eb4d1c87686e612bded898] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp, totalSize=33.1 K 2023-03-29 03:53:45,731 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 1cc6b52678be4b239ffa8879a819583f, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=8, earliestPutTs=1680062023450 2023-03-29 03:53:45,737 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 5fdd7a7a65c6418887a869fa3d363115, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=263, earliestPutTs=1680062024313 2023-03-29 03:53:45,740 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting b817ca9e28eb4d1c87686e612bded898, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=278, earliestPutTs=1680062024522 2023-03-29 03:53:45,776 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] throttle.PressureAwareThroughputController(145): 52c064cf4695189b45c25208ecad32f7#C#compaction#14 average throughput is 1.31 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:53:45,779 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:45,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48240 deadline: 1680062085778, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:45,784 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:53:45,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:53:45,801 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=296 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/C/1bf19174cb284d1db5eaac7279f1d336 2023-03-29 03:53:45,812 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:45,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48252 deadline: 1680062085789, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:45,813 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:45,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48262 deadline: 1680062085790, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:45,813 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:45,813 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:53:45,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48198 deadline: 1680062085791, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:45,815 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:45,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:53:45,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:53:45,821 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:53:45,821 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:53:45,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48182 deadline: 1680062085801, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:45,826 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:53:45,826 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:53:45,827 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:45,827 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:45,831 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:53:45,831 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:45,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:53:45,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:53:45,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:45,832 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:45,832 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:53:45,833 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:45,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:53:45,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:45,834 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:53:45,835 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/A/d4d0ac16fd8445fe94434c01ed615900 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/d4d0ac16fd8445fe94434c01ed615900 2023-03-29 03:53:45,840 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/C/4fe2ba50f7e8411d8ed85bfd4c2f6216 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/4fe2ba50f7e8411d8ed85bfd4c2f6216 2023-03-29 03:53:45,845 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/d4d0ac16fd8445fe94434c01ed615900, entries=200, sequenceid=296, filesize=14.3 K 2023-03-29 03:53:45,845 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:53:45,845 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:53:45,845 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:53:45,846 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:53:45,849 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:53:45,849 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:53:45,849 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:53:45,849 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:45,849 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. started... 2023-03-29 03:53:45,850 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. as already flushing 2023-03-29 03:53:45,850 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:45,850 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:53:45,850 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:53:45,850 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:45,850 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:53:45,850 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:53:45,853 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:45,853 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:45,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:45,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:45,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:45,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:45,854 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:45,854 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:53:45,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:45,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:45,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:45,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:45,856 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:53:45,856 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@cc39072[Count = 0] remaining members to acquire global barrier 2023-03-29 03:53:45,856 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:53:45,856 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:45,858 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:45,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:45,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:45,858 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:53:45,858 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:53:45,858 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:53:45,858 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:45,858 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:53:45,860 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:53:45,860 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:45,860 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:53:45,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:45,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:45,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:45,860 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:53:45,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:45,864 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/B/e9991746cacf48aba8504b1506edde5b as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/e9991746cacf48aba8504b1506edde5b 2023-03-29 03:53:45,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:45,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:45,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:45,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:45,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:45,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:45,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:53:45,877 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:53:45,877 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:53:45,877 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:53:45,877 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:53:45,877 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:53:45,877 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:53:45,885 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-03-29 03:53:45,894 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:45,894 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:53:45,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:45,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:45,894 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:45,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:45,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:45,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:45,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:53:45,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:53:45,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:45,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:45,902 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:45,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:45,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:45,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:45,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:45,904 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 52c064cf4695189b45c25208ecad32f7/C of 52c064cf4695189b45c25208ecad32f7 into 4fe2ba50f7e8411d8ed85bfd4c2f6216(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:53:45,904 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 52c064cf4695189b45c25208ecad32f7: 2023-03-29 03:53:45,904 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:45,904 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7., storeName=52c064cf4695189b45c25208ecad32f7/C, priority=13, startTime=1680062025512; duration=0sec 2023-03-29 03:53:45,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:45,905 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:53:45,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:45,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:45,912 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:53:45,912 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:53:45,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:53:45,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:53:45,912 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:45,912 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:45,912 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:45,912 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:45,912 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:53:45,912 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:45,921 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:45,921 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:45,921 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:53:45,922 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:53:45,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:53:45,922 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/e9991746cacf48aba8504b1506edde5b, entries=150, sequenceid=296, filesize=11.9 K 2023-03-29 03:53:45,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:53:45,922 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:53:45,922 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:53:45,923 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:53:45,924 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/C/1bf19174cb284d1db5eaac7279f1d336 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/1bf19174cb284d1db5eaac7279f1d336 2023-03-29 03:53:45,954 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/1bf19174cb284d1db5eaac7279f1d336, entries=150, sequenceid=296, filesize=11.9 K 2023-03-29 03:53:45,955 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=80.51 KB/82440 for 52c064cf4695189b45c25208ecad32f7 in 373ms, sequenceid=296, compaction requested=false 2023-03-29 03:53:45,955 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 52c064cf4695189b45c25208ecad32f7: 2023-03-29 03:53:45,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on 52c064cf4695189b45c25208ecad32f7 2023-03-29 03:53:45,990 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 52c064cf4695189b45c25208ecad32f7 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-03-29 03:53:45,990 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 52c064cf4695189b45c25208ecad32f7, store=A 2023-03-29 03:53:45,990 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:53:45,990 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 52c064cf4695189b45c25208ecad32f7, store=B 2023-03-29 03:53:45,990 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:53:45,990 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 52c064cf4695189b45c25208ecad32f7, store=C 2023-03-29 03:53:45,990 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:53:46,022 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:53:46,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:53:46,029 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:53:46,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:53:46,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:53:46,033 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:53:46,033 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:53:46,035 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:53:46,035 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:53:46,036 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:46,036 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:46,038 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-03-29 03:53:46,038 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:46,038 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:53:46,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:53:46,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:53:46,039 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:46,039 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:53:46,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:46,041 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-03-29 03:53:46,041 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:46,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:53:46,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:46,042 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:53:46,043 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:53:46,043 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:53:46,043 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:53:46,043 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:53:46,044 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:53:46,045 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:53:46,045 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:53:46,045 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:46,045 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. started... 2023-03-29 03:53:46,045 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. as already flushing 2023-03-29 03:53:46,045 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:46,045 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:53:46,045 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:53:46,045 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:46,045 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:53:46,045 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:53:46,049 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:46,049 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:46,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:46,050 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:46,050 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:53:46,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:46,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:46,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:46,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:46,058 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:46,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48198 deadline: 1680062086047, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:46,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:46,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:46,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:46,062 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:53:46,062 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ffcfc4d[Count = 0] remaining members to acquire global barrier 2023-03-29 03:53:46,062 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:53:46,062 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:46,064 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:46,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:46,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:46,065 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:53:46,065 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:53:46,065 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:53:46,065 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:46,065 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:53:46,068 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:53:46,068 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:53:46,068 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:53:46,073 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:46,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:46,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:46,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:46,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:46,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:46,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:46,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:46,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:46,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:46,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:46,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:53:46,077 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:53:46,077 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:53:46,078 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:53:46,078 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:53:46,078 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:53:46,078 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:53:46,079 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/B/3206fe95fbcf4b1abb047c143a5d762a as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/3206fe95fbcf4b1abb047c143a5d762a 2023-03-29 03:53:46,079 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:46,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48240 deadline: 1680062086054, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:46,080 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:46,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48182 deadline: 1680062086056, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:46,090 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:46,087 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:46,092 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:53:46,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:46,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:46,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:46,092 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:46,092 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:46,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48262 deadline: 1680062086059, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:46,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:46,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:46,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:46,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:53:46,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:53:46,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:46,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:46,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:46,094 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:46,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48252 deadline: 1680062086074, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:46,096 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:46,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:46,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:46,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:46,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:46,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:46,104 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:53:46,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:53:46,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:53:46,104 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:53:46,104 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:46,104 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:46,104 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:46,104 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:46,104 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:46,105 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:46,105 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:46,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:53:46,105 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:53:46,105 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:53:46,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:53:46,105 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:53:46,111 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:53:46,112 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:53:46,112 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:53:46,114 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 52c064cf4695189b45c25208ecad32f7/B of 52c064cf4695189b45c25208ecad32f7 into 3206fe95fbcf4b1abb047c143a5d762a(size=12.0 K), total size for store is 23.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:53:46,116 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 52c064cf4695189b45c25208ecad32f7: 2023-03-29 03:53:46,116 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7., storeName=52c064cf4695189b45c25208ecad32f7/B, priority=13, startTime=1680062025511; duration=0sec 2023-03-29 03:53:46,116 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:53:46,162 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:46,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48198 deadline: 1680062086161, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:46,194 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:46,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48240 deadline: 1680062086192, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:46,194 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:46,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48182 deadline: 1680062086194, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:46,196 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:46,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48262 deadline: 1680062086196, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:46,201 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:46,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48252 deadline: 1680062086198, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:46,212 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:53:46,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:53:46,217 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:53:46,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:53:46,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:53:46,226 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:53:46,226 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:53:46,228 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:53:46,228 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:53:46,229 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:46,229 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:46,235 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:53:46,235 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:46,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:53:46,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:53:46,235 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:46,235 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:53:46,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:46,236 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:46,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:53:46,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:46,237 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:53:46,244 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:53:46,244 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:53:46,244 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:53:46,244 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:53:46,250 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:53:46,254 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:53:46,254 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:53:46,254 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:46,254 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. started... 2023-03-29 03:53:46,254 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. as already flushing 2023-03-29 03:53:46,254 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:46,254 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:53:46,254 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:53:46,254 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:46,254 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:53:46,255 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:53:46,261 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:46,261 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:46,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:46,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:46,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:46,261 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:46,261 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:53:46,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:46,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:46,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:46,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:46,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:46,264 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:53:46,264 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b39db26[Count = 0] remaining members to acquire global barrier 2023-03-29 03:53:46,264 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:53:46,264 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:46,265 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:46,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:46,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:46,265 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:53:46,266 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:53:46,266 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:53:46,266 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:46,266 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:53:46,267 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:46,267 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:53:46,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:46,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:46,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:46,268 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:53:46,268 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:53:46,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:46,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:46,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:46,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:46,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:46,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:46,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:46,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:53:46,283 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:53:46,283 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:53:46,283 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:53:46,283 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:53:46,283 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:53:46,283 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:53:46,285 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:46,285 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:46,285 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:53:46,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:46,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:46,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:46,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:46,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:46,286 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:46,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:53:46,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:53:46,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:46,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:46,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:46,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:46,289 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:46,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:46,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:46,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:46,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:46,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:46,294 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:46,294 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:46,294 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:46,294 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:46,294 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:46,294 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:46,294 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:53:46,294 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:46,294 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:53:46,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:53:46,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:53:46,294 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:53:46,295 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:53:46,294 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:53:46,296 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:53:46,296 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:53:46,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:53:46,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:53:46,296 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:53:46,370 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:46,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48198 deadline: 1680062086366, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:46,396 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:53:46,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:53:46,402 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:53:46,404 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:46,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48240 deadline: 1680062086397, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:46,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:53:46,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:53:46,404 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:53:46,404 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:53:46,404 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:46,405 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:46,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48262 deadline: 1680062086399, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:46,406 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:53:46,406 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:53:46,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48182 deadline: 1680062086398, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:46,407 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:46,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48252 deadline: 1680062086404, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:46,407 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:46,407 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:46,409 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:46,409 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:53:46,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:53:46,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:53:46,410 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:46,410 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:53:46,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:46,411 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:46,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:53:46,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:46,412 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:53:46,412 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:53:46,412 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:53:46,412 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:53:46,412 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:53:46,417 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:53:46,417 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:53:46,417 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:53:46,417 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:46,417 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. started... 2023-03-29 03:53:46,417 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. as already flushing 2023-03-29 03:53:46,417 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:46,417 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:53:46,417 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:53:46,417 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:46,417 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:53:46,418 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:53:46,420 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:46,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:46,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:46,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:46,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:46,421 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:46,421 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:46,421 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:53:46,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:46,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:46,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:46,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:46,423 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:53:46,423 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7fbde927[Count = 0] remaining members to acquire global barrier 2023-03-29 03:53:46,423 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:53:46,423 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:46,425 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:46,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:46,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:46,425 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:53:46,425 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:53:46,425 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:53:46,425 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:46,426 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:53:46,429 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:53:46,429 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:53:46,429 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:46,429 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:53:46,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:46,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:46,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:46,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:46,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:46,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:46,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:46,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:46,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:46,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:46,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:53:46,437 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:53:46,437 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:53:46,437 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:53:46,437 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:53:46,437 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:53:46,437 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:53:46,440 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:46,440 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:53:46,440 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:46,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:46,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:46,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:46,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:46,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:46,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:53:46,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:53:46,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:46,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:46,442 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:46,444 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=314 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/A/d620eee792fb4f95a94cfa6f8b77c04f 2023-03-29 03:53:46,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:46,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:46,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:46,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:46,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:46,448 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:46,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:46,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:46,455 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:53:46,455 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:53:46,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:53:46,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:53:46,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:53:46,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:53:46,457 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:46,457 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:46,457 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:46,457 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:53:46,457 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:46,457 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:46,457 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:46,457 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:46,457 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:53:46,457 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:53:46,457 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:53:46,458 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:53:46,502 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=314 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/B/75b82637332441feb0865fefbb7a593b 2023-03-29 03:53:46,558 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:53:46,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:53:46,563 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:53:46,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:53:46,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:53:46,570 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:53:46,570 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:53:46,573 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:53:46,573 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:53:46,573 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:46,574 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:46,575 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:53:46,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:53:46,575 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:46,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:53:46,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:46,577 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:46,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:53:46,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:46,578 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:53:46,578 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:46,578 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:53:46,578 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:53:46,578 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:53:46,578 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:53:46,578 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:53:46,580 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:53:46,583 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:53:46,583 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:53:46,583 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:46,583 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. started... 2023-03-29 03:53:46,584 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. as already flushing 2023-03-29 03:53:46,584 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:46,584 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:53:46,584 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:53:46,584 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:46,584 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:53:46,584 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:53:46,591 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:46,591 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:46,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:46,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:46,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:46,592 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:46,592 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:53:46,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:46,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:46,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:46,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:46,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:46,594 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:53:46,594 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f6db6a2[Count = 0] remaining members to acquire global barrier 2023-03-29 03:53:46,594 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:53:46,595 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:46,596 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:46,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:46,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:46,596 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:53:46,596 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:53:46,596 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:53:46,597 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:46,597 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:53:46,598 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:53:46,598 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:53:46,598 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:53:46,600 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:46,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:46,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:46,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:46,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:46,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:46,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:46,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:46,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:46,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:46,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:46,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:53:46,604 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:53:46,604 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:53:46,604 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:53:46,604 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:53:46,604 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:53:46,604 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:53:46,608 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:46,608 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:46,608 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:53:46,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:46,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:46,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:46,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:46,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:46,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:53:46,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:53:46,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:46,610 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:46,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:46,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:46,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:46,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:46,612 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:46,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:46,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:46,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:46,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:46,615 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:53:46,615 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:46,615 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:53:46,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:53:46,615 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:53:46,615 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:53:46,615 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:46,615 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:46,615 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:46,616 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:46,616 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:53:46,616 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:53:46,616 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:53:46,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:53:46,615 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:53:46,616 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:46,616 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:46,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:53:46,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:53:46,675 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:46,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48198 deadline: 1680062086674, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:46,710 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:46,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48240 deadline: 1680062086708, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:46,716 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:53:46,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:53:46,717 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:46,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48262 deadline: 1680062086710, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:46,718 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:46,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48252 deadline: 1680062086711, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:46,719 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:46,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48182 deadline: 1680062086711, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:46,721 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:53:46,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:53:46,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:53:46,730 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:53:46,730 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:53:46,733 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:53:46,733 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:53:46,733 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:46,734 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:46,735 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:53:46,735 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:46,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:53:46,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:53:46,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:46,736 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:46,736 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:53:46,736 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:46,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:53:46,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:46,739 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:53:46,740 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:53:46,740 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:53:46,740 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:53:46,740 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:53:46,745 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:53:46,745 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:53:46,745 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:53:46,745 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:46,745 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. started... 2023-03-29 03:53:46,745 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. as already flushing 2023-03-29 03:53:46,745 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:46,745 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:53:46,746 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:53:46,746 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:46,746 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:53:46,746 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:53:46,747 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:46,747 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:46,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:46,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:46,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:46,748 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:46,748 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:53:46,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:46,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:46,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:46,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:46,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:46,751 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:53:46,751 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@273b3196[Count = 0] remaining members to acquire global barrier 2023-03-29 03:53:46,751 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:53:46,751 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:46,753 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:46,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:46,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:46,754 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:53:46,754 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:53:46,754 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:53:46,754 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:46,754 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:53:46,756 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:53:46,756 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:53:46,756 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:53:46,757 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:46,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:46,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:46,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:46,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:46,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:46,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:46,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:46,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:46,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:46,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:46,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:53:46,765 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:53:46,765 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:53:46,765 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:53:46,765 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:53:46,765 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:53:46,765 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:53:46,767 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:46,767 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:53:46,767 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:46,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:46,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:46,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:46,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:46,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:46,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:53:46,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:53:46,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:46,769 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:46,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:46,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:46,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:46,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:46,771 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:46,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:46,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:46,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:46,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:46,775 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:53:46,775 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:53:46,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:53:46,775 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:46,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:53:46,775 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:53:46,775 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:46,775 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:53:46,775 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:53:46,775 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:46,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:53:46,776 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:46,776 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:46,776 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:46,777 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:53:46,777 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:46,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:53:46,777 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:53:46,877 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:53:46,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:53:46,881 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:53:46,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:53:46,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:53:46,888 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:53:46,888 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:53:46,890 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:53:46,890 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:53:46,890 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:46,890 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:46,892 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:53:46,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:53:46,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:53:46,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:46,893 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:46,894 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:46,894 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:46,894 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:53:46,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:53:46,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:46,894 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:53:46,896 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:53:46,896 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:53:46,896 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:53:46,896 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:53:46,905 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:53:46,905 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:53:46,905 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:53:46,905 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:46,919 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. started... 2023-03-29 03:53:46,919 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. as already flushing 2023-03-29 03:53:46,919 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:46,919 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:53:46,919 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:53:46,919 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:46,920 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:53:46,920 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:53:46,922 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:46,922 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:46,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:46,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:46,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:46,922 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:46,922 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:53:46,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:46,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:46,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:46,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:46,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:46,924 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:53:46,924 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:53:46,924 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@12ee3ebc[Count = 0] remaining members to acquire global barrier 2023-03-29 03:53:46,925 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:46,926 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:46,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:46,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:46,927 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:46,927 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:53:46,927 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:53:46,927 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:53:46,927 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:53:46,930 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:53:46,930 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:46,930 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:53:46,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:46,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:46,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:46,930 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:53:46,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:46,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:46,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:46,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:46,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:46,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:46,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:46,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:53:46,938 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:53:46,939 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:53:46,939 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:53:46,939 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:53:46,939 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:53:46,939 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:53:46,941 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:46,941 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:53:46,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:46,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:46,941 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:46,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:46,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:46,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:46,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:53:46,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:53:46,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:46,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:46,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:46,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:46,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:46,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:46,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:46,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:46,945 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:46,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:46,949 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:46,952 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:53:46,953 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:46,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:53:46,953 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:53:46,953 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:53:46,957 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:53:46,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:53:46,953 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:46,953 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:53:46,957 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:46,957 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:53:46,957 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:53:46,957 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:46,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:53:46,958 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:46,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:53:46,958 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:46,958 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:46,958 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=314 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/C/78fe1506bd2940d7b34ef99e2f3121e0 2023-03-29 03:53:46,972 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/A/d620eee792fb4f95a94cfa6f8b77c04f as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/d620eee792fb4f95a94cfa6f8b77c04f 2023-03-29 03:53:46,996 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/d620eee792fb4f95a94cfa6f8b77c04f, entries=150, sequenceid=314, filesize=11.9 K 2023-03-29 03:53:47,004 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/B/75b82637332441feb0865fefbb7a593b as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/75b82637332441feb0865fefbb7a593b 2023-03-29 03:53:47,022 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/75b82637332441feb0865fefbb7a593b, entries=150, sequenceid=314, filesize=11.9 K 2023-03-29 03:53:47,025 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/C/78fe1506bd2940d7b34ef99e2f3121e0 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/78fe1506bd2940d7b34ef99e2f3121e0 2023-03-29 03:53:47,041 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/78fe1506bd2940d7b34ef99e2f3121e0, entries=150, sequenceid=314, filesize=11.9 K 2023-03-29 03:53:47,046 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=93.93 KB/96180 for 52c064cf4695189b45c25208ecad32f7 in 1056ms, sequenceid=314, compaction requested=true 2023-03-29 03:53:47,046 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 52c064cf4695189b45c25208ecad32f7: 2023-03-29 03:53:47,046 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:53:47,046 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-29 03:53:47,046 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-29 03:53:47,046 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:53:47,047 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:53:47,049 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39101 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-29 03:53:47,049 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36661 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-29 03:53:47,049 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1912): 52c064cf4695189b45c25208ecad32f7/A is initiating minor compaction (all files) 2023-03-29 03:53:47,049 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1912): 52c064cf4695189b45c25208ecad32f7/B is initiating minor compaction (all files) 2023-03-29 03:53:47,049 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 52c064cf4695189b45c25208ecad32f7/A in TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:47,049 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 52c064cf4695189b45c25208ecad32f7/B in TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:47,049 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/ea13c770aa4b4756b2fb2465c8878fe4, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/d4d0ac16fd8445fe94434c01ed615900, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/d620eee792fb4f95a94cfa6f8b77c04f] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp, totalSize=38.2 K 2023-03-29 03:53:47,050 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/3206fe95fbcf4b1abb047c143a5d762a, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/e9991746cacf48aba8504b1506edde5b, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/75b82637332441feb0865fefbb7a593b] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp, totalSize=35.8 K 2023-03-29 03:53:47,050 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting ea13c770aa4b4756b2fb2465c8878fe4, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=278, earliestPutTs=1680062024522 2023-03-29 03:53:47,050 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 3206fe95fbcf4b1abb047c143a5d762a, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=278, earliestPutTs=1680062024522 2023-03-29 03:53:47,053 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting d4d0ac16fd8445fe94434c01ed615900, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=296, earliestPutTs=1680062025242 2023-03-29 03:53:47,053 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting e9991746cacf48aba8504b1506edde5b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=296, earliestPutTs=1680062025245 2023-03-29 03:53:47,054 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting d620eee792fb4f95a94cfa6f8b77c04f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=314, earliestPutTs=1680062025638 2023-03-29 03:53:47,055 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 75b82637332441feb0865fefbb7a593b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=314, earliestPutTs=1680062025638 2023-03-29 03:53:47,058 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:53:47,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:53:47,064 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:53:47,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:53:47,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:53:47,079 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:53:47,079 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:53:47,083 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:53:47,083 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:53:47,084 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:47,084 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:47,086 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] throttle.PressureAwareThroughputController(145): 52c064cf4695189b45c25208ecad32f7#A#compaction#19 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:53:47,086 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:53:47,086 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] throttle.PressureAwareThroughputController(145): 52c064cf4695189b45c25208ecad32f7#B#compaction#18 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:53:47,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:53:47,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:53:47,086 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:47,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:47,087 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:47,087 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:53:47,088 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:47,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:53:47,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:47,088 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:53:47,092 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:53:47,092 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:53:47,092 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:53:47,092 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:53:47,093 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:53:47,093 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:53:47,093 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:53:47,093 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:47,093 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. started... 2023-03-29 03:53:47,094 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 52c064cf4695189b45c25208ecad32f7 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-03-29 03:53:47,094 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 52c064cf4695189b45c25208ecad32f7, store=A 2023-03-29 03:53:47,094 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:53:47,094 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 52c064cf4695189b45c25208ecad32f7, store=B 2023-03-29 03:53:47,094 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:53:47,094 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 52c064cf4695189b45c25208ecad32f7, store=C 2023-03-29 03:53:47,094 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:53:47,166 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=332 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/A/2482f46a69964531849535552193bf44 2023-03-29 03:53:47,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on 52c064cf4695189b45c25208ecad32f7 2023-03-29 03:53:47,182 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. as already flushing 2023-03-29 03:53:47,210 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=332 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/B/2644b2b874074b5a890ea09af5fc0ed6 2023-03-29 03:53:47,239 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:47,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48182 deadline: 1680062087233, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:47,240 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:47,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48262 deadline: 1680062087235, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:47,240 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:47,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48240 deadline: 1680062087237, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:47,242 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:47,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48198 deadline: 1680062087239, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:47,242 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:47,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48252 deadline: 1680062087239, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:47,261 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=332 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/C/ce816c8bd0bc442e9bb8155e8082649c 2023-03-29 03:53:47,273 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/A/2482f46a69964531849535552193bf44 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/2482f46a69964531849535552193bf44 2023-03-29 03:53:47,283 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/2482f46a69964531849535552193bf44, entries=150, sequenceid=332, filesize=11.9 K 2023-03-29 03:53:47,286 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/B/2644b2b874074b5a890ea09af5fc0ed6 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/2644b2b874074b5a890ea09af5fc0ed6 2023-03-29 03:53:47,297 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/2644b2b874074b5a890ea09af5fc0ed6, entries=150, sequenceid=332, filesize=11.9 K 2023-03-29 03:53:47,300 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/C/ce816c8bd0bc442e9bb8155e8082649c as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/ce816c8bd0bc442e9bb8155e8082649c 2023-03-29 03:53:47,325 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/ce816c8bd0bc442e9bb8155e8082649c, entries=150, sequenceid=332, filesize=11.9 K 2023-03-29 03:53:47,329 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=87.22 KB/89310 for 52c064cf4695189b45c25208ecad32f7 in 232ms, sequenceid=332, compaction requested=true 2023-03-29 03:53:47,329 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 52c064cf4695189b45c25208ecad32f7: 2023-03-29 03:53:47,329 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:47,329 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:53:47,331 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:53:47,331 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:47,331 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:53:47,331 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:53:47,333 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:47,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:47,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:47,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:47,334 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:47,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:47,334 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:47,334 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:53:47,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:47,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:47,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:47,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:47,336 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:53:47,336 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@20a1917[Count = 0] remaining members to acquire global barrier 2023-03-29 03:53:47,336 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:53:47,337 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:47,340 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:47,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:47,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:47,341 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:53:47,341 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:53:47,341 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:53:47,341 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:47,341 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:53:47,343 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:53:47,343 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:53:47,343 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:53:47,343 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:47,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:47,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:47,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:47,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:47,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:47,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:47,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:47,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:47,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on 52c064cf4695189b45c25208ecad32f7 2023-03-29 03:53:47,352 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 52c064cf4695189b45c25208ecad32f7 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-03-29 03:53:47,353 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 52c064cf4695189b45c25208ecad32f7, store=A 2023-03-29 03:53:47,353 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:53:47,353 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 52c064cf4695189b45c25208ecad32f7, store=B 2023-03-29 03:53:47,353 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:53:47,353 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 52c064cf4695189b45c25208ecad32f7, store=C 2023-03-29 03:53:47,353 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:53:47,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:47,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:47,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:53:47,357 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:53:47,357 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:53:47,357 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:53:47,357 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:53:47,357 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:53:47,357 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:53:47,360 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:47,360 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:47,360 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:53:47,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:47,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:47,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:53:47,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:53:47,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:47,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:47,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:47,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:47,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:47,364 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:47,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:47,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:47,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:47,366 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:47,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:47,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:47,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:47,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:47,371 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:47,371 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:53:47,371 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:53:47,371 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:47,371 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:53:47,371 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:47,371 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:53:47,372 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:47,372 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:53:47,374 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:47,376 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:47,376 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:47,373 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:53:47,376 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:53:47,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:53:47,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:53:47,376 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:53:47,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:53:47,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:53:47,383 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:47,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48252 deadline: 1680062087377, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:47,384 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:47,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48262 deadline: 1680062087382, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:47,401 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:47,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48182 deadline: 1680062087382, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:47,405 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:47,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48240 deadline: 1680062087383, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:47,406 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:47,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48198 deadline: 1680062087384, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:47,467 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(130): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-03-29 03:53:47,468 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2023-03-29 03:53:47,476 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:53:47,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:53:47,481 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:53:47,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:53:47,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:53:47,483 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:53:47,483 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:53:47,484 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:53:47,484 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:53:47,484 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:47,485 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:47,490 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:47,491 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:53:47,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:53:47,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:53:47,491 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:47,491 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:53:47,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:47,494 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:47,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:53:47,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:47,495 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:53:47,495 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:53:47,495 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:53:47,495 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:53:47,497 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:53:47,511 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:47,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48252 deadline: 1680062087502, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:47,511 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:47,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48262 deadline: 1680062087502, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:47,512 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:47,514 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:47,512 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:47,512 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:47,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48240 deadline: 1680062087507, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:47,512 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:53:47,514 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. started... 2023-03-29 03:53:47,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48198 deadline: 1680062087509, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:47,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48182 deadline: 1680062087502, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:47,514 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. as already flushing 2023-03-29 03:53:47,514 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:53:47,515 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:47,515 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:53:47,515 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:53:47,515 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:47,515 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:53:47,515 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:53:47,517 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:47,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:47,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:47,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:47,518 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:47,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:47,518 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:47,518 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:53:47,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:47,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:47,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:47,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:47,520 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:53:47,520 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@d06752[Count = 0] remaining members to acquire global barrier 2023-03-29 03:53:47,520 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:53:47,520 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:47,522 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:47,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:47,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:47,522 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:53:47,522 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:53:47,522 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:53:47,523 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:47,523 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:53:47,524 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:53:47,524 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:47,524 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:53:47,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:47,524 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:53:47,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:47,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:47,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:47,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:47,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:47,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:47,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:47,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:47,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:47,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:53:47,529 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:53:47,529 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:53:47,529 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:53:47,529 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:53:47,529 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:53:47,529 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:53:47,533 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:47,533 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:53:47,533 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:47,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:47,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:47,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:47,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:47,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:47,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:53:47,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:53:47,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:47,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:47,535 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:47,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:47,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:47,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:47,536 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:47,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:47,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:47,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:47,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:47,541 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:47,541 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:53:47,541 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:53:47,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:53:47,541 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:53:47,541 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:47,541 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:47,541 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:47,541 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:47,541 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:47,541 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:47,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:53:47,541 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:53:47,541 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:53:47,543 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:53:47,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:53:47,543 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:53:47,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:53:47,543 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:53:47,571 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/A/c576cefc7c484428a0e43f77953a0e76 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/c576cefc7c484428a0e43f77953a0e76 2023-03-29 03:53:47,573 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/B/d3052abb90da48ba928d22c36d572ac5 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/d3052abb90da48ba928d22c36d572ac5 2023-03-29 03:53:47,584 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 52c064cf4695189b45c25208ecad32f7/B of 52c064cf4695189b45c25208ecad32f7 into d3052abb90da48ba928d22c36d572ac5(size=12.1 K), total size for store is 24.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:53:47,584 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 52c064cf4695189b45c25208ecad32f7: 2023-03-29 03:53:47,584 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7., storeName=52c064cf4695189b45c25208ecad32f7/B, priority=13, startTime=1680062027046; duration=0sec 2023-03-29 03:53:47,585 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:53:47,585 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-29 03:53:47,588 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 52c064cf4695189b45c25208ecad32f7/A of 52c064cf4695189b45c25208ecad32f7 into c576cefc7c484428a0e43f77953a0e76(size=12.1 K), total size for store is 24.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:53:47,589 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 52c064cf4695189b45c25208ecad32f7: 2023-03-29 03:53:47,589 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48847 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-29 03:53:47,589 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1912): 52c064cf4695189b45c25208ecad32f7/C is initiating minor compaction (all files) 2023-03-29 03:53:47,589 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7., storeName=52c064cf4695189b45c25208ecad32f7/A, priority=13, startTime=1680062027046; duration=0sec 2023-03-29 03:53:47,589 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 52c064cf4695189b45c25208ecad32f7/C in TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:47,589 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:53:47,592 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/4fe2ba50f7e8411d8ed85bfd4c2f6216, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/1bf19174cb284d1db5eaac7279f1d336, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/78fe1506bd2940d7b34ef99e2f3121e0, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/ce816c8bd0bc442e9bb8155e8082649c] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp, totalSize=47.7 K 2023-03-29 03:53:47,593 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 4fe2ba50f7e8411d8ed85bfd4c2f6216, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=278, earliestPutTs=1680062024522 2023-03-29 03:53:47,594 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 1bf19174cb284d1db5eaac7279f1d336, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=296, earliestPutTs=1680062025245 2023-03-29 03:53:47,595 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 78fe1506bd2940d7b34ef99e2f3121e0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=314, earliestPutTs=1680062025638 2023-03-29 03:53:47,598 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting ce816c8bd0bc442e9bb8155e8082649c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=332, earliestPutTs=1680062026043 2023-03-29 03:53:47,616 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] throttle.PressureAwareThroughputController(145): 52c064cf4695189b45c25208ecad32f7#C#compaction#24 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:53:47,643 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:53:47,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:53:47,655 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/C/a2d935a73df840608cacf3c3c925d769 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/a2d935a73df840608cacf3c3c925d769 2023-03-29 03:53:47,656 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:53:47,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:53:47,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:53:47,667 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:53:47,667 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:53:47,668 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:53:47,668 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:53:47,668 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 52c064cf4695189b45c25208ecad32f7/C of 52c064cf4695189b45c25208ecad32f7 into a2d935a73df840608cacf3c3c925d769(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:53:47,668 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 52c064cf4695189b45c25208ecad32f7: 2023-03-29 03:53:47,669 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7., storeName=52c064cf4695189b45c25208ecad32f7/C, priority=12, startTime=1680062027047; duration=0sec 2023-03-29 03:53:47,669 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:53:47,669 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:47,669 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:47,672 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:53:47,672 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:47,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:53:47,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:53:47,672 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:47,672 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:53:47,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:47,673 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:47,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:53:47,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:47,673 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:53:47,673 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:53:47,673 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:53:47,676 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:53:47,676 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:53:47,676 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:53:47,677 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:53:47,677 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:53:47,677 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:47,677 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. started... 2023-03-29 03:53:47,677 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. as already flushing 2023-03-29 03:53:47,677 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:47,677 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:53:47,677 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:53:47,677 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:47,677 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:53:47,677 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:53:47,679 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:47,679 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:47,679 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:53:47,679 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:47,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:47,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:47,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:47,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:47,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:47,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:47,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:47,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:47,682 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:53:47,682 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7e99515f[Count = 0] remaining members to acquire global barrier 2023-03-29 03:53:47,682 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:53:47,682 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:47,685 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:47,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:47,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:47,685 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:53:47,685 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:53:47,685 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:53:47,686 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:47,686 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:53:47,686 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:53:47,687 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:47,687 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:53:47,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:47,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:47,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:47,687 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:53:47,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:47,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:47,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:47,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:47,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:47,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:47,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:47,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:53:47,692 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:53:47,692 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:53:47,692 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:53:47,692 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:53:47,692 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:53:47,692 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:53:47,695 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:47,695 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:47,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:47,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:47,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:47,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:47,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:47,695 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:53:47,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:53:47,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:53:47,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:47,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:47,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:47,697 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:47,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:47,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:47,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:47,700 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:47,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:47,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:47,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:47,703 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:47,703 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:47,703 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:47,703 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:47,703 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:47,703 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:47,703 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:47,703 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:53:47,703 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:53:47,704 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:53:47,704 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:53:47,704 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:53:47,703 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:53:47,704 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:53:47,706 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:53:47,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:53:47,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:53:47,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:53:47,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:53:47,714 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:47,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48262 deadline: 1680062087713, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:47,716 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:47,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48252 deadline: 1680062087714, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:47,717 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:47,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48240 deadline: 1680062087717, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:47,718 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:47,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48182 deadline: 1680062087718, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:47,721 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:47,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48198 deadline: 1680062087719, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:47,784 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=349 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/A/de45383171e342dfbb02eb813adc7f86 2023-03-29 03:53:47,805 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:53:47,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:53:47,811 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:53:47,812 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=349 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/B/027d19c162174e6b91142b3b7ab542f2 2023-03-29 03:53:47,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:53:47,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:53:47,814 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:53:47,814 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:53:47,815 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:53:47,815 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:53:47,816 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:47,816 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:47,818 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:53:47,818 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:47,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:53:47,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:53:47,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:47,819 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:47,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:53:47,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:47,820 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:53:47,820 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:47,820 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:53:47,820 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:53:47,820 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:53:47,821 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:53:47,821 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:53:47,824 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:53:47,825 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:53:47,825 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:53:47,825 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:47,825 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. started... 2023-03-29 03:53:47,826 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. as already flushing 2023-03-29 03:53:47,826 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:47,826 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:53:47,826 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:53:47,826 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:47,826 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:53:47,826 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:53:47,828 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:47,828 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:47,828 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:53:47,831 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:47,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:47,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:47,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:47,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:47,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:47,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:47,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:47,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:47,835 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:53:47,835 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1de18bee[Count = 0] remaining members to acquire global barrier 2023-03-29 03:53:47,835 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:53:47,835 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:47,837 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:47,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:47,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:47,841 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:53:47,841 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:53:47,841 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:47,841 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:53:47,841 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:53:47,843 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=349 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/C/d62336621bae428683f69cf215e02184 2023-03-29 03:53:47,843 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:53:47,843 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:47,844 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:53:47,844 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:53:47,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:47,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:47,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:47,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:47,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:47,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:47,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:47,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:47,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:47,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:47,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:53:47,848 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:53:47,848 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:53:47,848 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:53:47,848 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:53:47,849 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:53:47,849 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:53:47,853 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/A/de45383171e342dfbb02eb813adc7f86 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/de45383171e342dfbb02eb813adc7f86 2023-03-29 03:53:47,869 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:47,869 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:47,869 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:53:47,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:47,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:47,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:47,869 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/de45383171e342dfbb02eb813adc7f86, entries=200, sequenceid=349, filesize=14.3 K 2023-03-29 03:53:47,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:47,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:47,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:53:47,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:53:47,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:47,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:47,875 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:47,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:47,875 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/B/027d19c162174e6b91142b3b7ab542f2 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/027d19c162174e6b91142b3b7ab542f2 2023-03-29 03:53:47,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:47,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:47,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:47,878 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:47,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:47,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:47,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:47,883 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:47,883 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:47,883 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:47,883 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:47,883 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:47,883 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:53:47,883 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:47,883 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:47,883 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:53:47,883 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:53:47,883 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:53:47,883 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:53:47,884 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:53:47,885 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:53:47,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:53:47,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:53:47,885 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:53:47,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:53:47,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:53:47,894 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/027d19c162174e6b91142b3b7ab542f2, entries=150, sequenceid=349, filesize=11.9 K 2023-03-29 03:53:47,896 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/C/d62336621bae428683f69cf215e02184 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/d62336621bae428683f69cf215e02184 2023-03-29 03:53:47,909 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/d62336621bae428683f69cf215e02184, entries=150, sequenceid=349, filesize=11.9 K 2023-03-29 03:53:47,913 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=87.22 KB/89310 for 52c064cf4695189b45c25208ecad32f7 in 561ms, sequenceid=349, compaction requested=true 2023-03-29 03:53:47,913 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 52c064cf4695189b45c25208ecad32f7: 2023-03-29 03:53:47,913 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:53:47,913 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-29 03:53:47,913 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:53:47,913 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-29 03:53:47,913 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:53:47,916 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39204 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-29 03:53:47,916 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1912): 52c064cf4695189b45c25208ecad32f7/A is initiating minor compaction (all files) 2023-03-29 03:53:47,916 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 52c064cf4695189b45c25208ecad32f7/A in TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:47,916 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/c576cefc7c484428a0e43f77953a0e76, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/2482f46a69964531849535552193bf44, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/de45383171e342dfbb02eb813adc7f86] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp, totalSize=38.3 K 2023-03-29 03:53:47,917 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting c576cefc7c484428a0e43f77953a0e76, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=314, earliestPutTs=1680062025638 2023-03-29 03:53:47,918 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 2482f46a69964531849535552193bf44, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=332, earliestPutTs=1680062026043 2023-03-29 03:53:47,918 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36764 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-29 03:53:47,919 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1912): 52c064cf4695189b45c25208ecad32f7/B is initiating minor compaction (all files) 2023-03-29 03:53:47,919 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 52c064cf4695189b45c25208ecad32f7/B in TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:47,919 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/d3052abb90da48ba928d22c36d572ac5, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/2644b2b874074b5a890ea09af5fc0ed6, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/027d19c162174e6b91142b3b7ab542f2] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp, totalSize=35.9 K 2023-03-29 03:53:47,920 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting de45383171e342dfbb02eb813adc7f86, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=349, earliestPutTs=1680062027234 2023-03-29 03:53:47,920 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting d3052abb90da48ba928d22c36d572ac5, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=314, earliestPutTs=1680062025638 2023-03-29 03:53:47,923 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 2644b2b874074b5a890ea09af5fc0ed6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=332, earliestPutTs=1680062026043 2023-03-29 03:53:47,924 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 027d19c162174e6b91142b3b7ab542f2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=349, earliestPutTs=1680062027235 2023-03-29 03:53:47,958 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] throttle.PressureAwareThroughputController(145): 52c064cf4695189b45c25208ecad32f7#A#compaction#27 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:53:47,958 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] throttle.PressureAwareThroughputController(145): 52c064cf4695189b45c25208ecad32f7#B#compaction#28 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:53:47,985 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:53:47,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:53:47,989 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:53:47,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:53:47,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:53:47,993 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:53:47,993 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:53:47,994 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:53:47,994 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:53:47,995 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:47,995 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:47,996 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:53:47,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:53:47,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:53:47,997 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:47,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:48,000 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:48,000 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:53:48,001 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:48,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:53:48,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:48,001 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:53:48,004 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:53:48,004 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:53:48,004 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:53:48,004 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:53:48,005 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:53:48,005 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:53:48,006 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:53:48,006 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:48,006 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. started... 2023-03-29 03:53:48,006 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 52c064cf4695189b45c25208ecad32f7 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-03-29 03:53:48,006 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 52c064cf4695189b45c25208ecad32f7, store=A 2023-03-29 03:53:48,006 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:53:48,006 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 52c064cf4695189b45c25208ecad32f7, store=B 2023-03-29 03:53:48,007 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:53:48,007 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 52c064cf4695189b45c25208ecad32f7, store=C 2023-03-29 03:53:48,007 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:53:48,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on 52c064cf4695189b45c25208ecad32f7 2023-03-29 03:53:48,022 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. as already flushing 2023-03-29 03:53:48,033 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=368 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/A/2acbca75782d454daf0cffd810199e73 2023-03-29 03:53:48,042 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:48,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48252 deadline: 1680062088038, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:48,044 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:48,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48262 deadline: 1680062088040, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:48,046 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:48,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48198 deadline: 1680062088041, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:48,047 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:48,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48182 deadline: 1680062088043, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:48,048 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:48,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48240 deadline: 1680062088043, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:48,073 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=368 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/B/67e255c7e47d4eeeaac0edda435f2968 2023-03-29 03:53:48,093 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=368 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/C/d116f3bf259446dfa2c77b7b983364e3 2023-03-29 03:53:48,101 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/A/2acbca75782d454daf0cffd810199e73 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/2acbca75782d454daf0cffd810199e73 2023-03-29 03:53:48,111 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/2acbca75782d454daf0cffd810199e73, entries=150, sequenceid=368, filesize=11.9 K 2023-03-29 03:53:48,113 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/B/67e255c7e47d4eeeaac0edda435f2968 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/67e255c7e47d4eeeaac0edda435f2968 2023-03-29 03:53:48,138 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/67e255c7e47d4eeeaac0edda435f2968, entries=150, sequenceid=368, filesize=11.9 K 2023-03-29 03:53:48,142 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/C/d116f3bf259446dfa2c77b7b983364e3 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/d116f3bf259446dfa2c77b7b983364e3 2023-03-29 03:53:48,151 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:48,151 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/d116f3bf259446dfa2c77b7b983364e3, entries=150, sequenceid=368, filesize=11.9 K 2023-03-29 03:53:48,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48252 deadline: 1680062088146, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:48,155 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=93.93 KB/96180 for 52c064cf4695189b45c25208ecad32f7 in 149ms, sequenceid=368, compaction requested=true 2023-03-29 03:53:48,155 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 52c064cf4695189b45c25208ecad32f7: 2023-03-29 03:53:48,155 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:48,156 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:53:48,156 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:53:48,156 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:48,163 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:53:48,163 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:53:48,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on 52c064cf4695189b45c25208ecad32f7 2023-03-29 03:53:48,164 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 52c064cf4695189b45c25208ecad32f7 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-03-29 03:53:48,164 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 52c064cf4695189b45c25208ecad32f7, store=A 2023-03-29 03:53:48,164 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:53:48,164 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 52c064cf4695189b45c25208ecad32f7, store=B 2023-03-29 03:53:48,164 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:53:48,164 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 52c064cf4695189b45c25208ecad32f7, store=C 2023-03-29 03:53:48,164 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:53:48,166 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:48,166 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:48,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:48,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:48,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:48,167 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:48,167 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:53:48,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:48,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:48,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:48,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:48,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:48,171 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:53:48,172 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f71fbe4[Count = 0] remaining members to acquire global barrier 2023-03-29 03:53:48,172 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:53:48,172 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:48,173 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:48,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:48,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:48,173 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:53:48,174 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:48,174 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:53:48,174 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:53:48,174 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:53:48,176 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:53:48,176 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:53:48,176 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:53:48,177 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:48,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:48,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:48,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:48,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:48,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:48,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:48,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:48,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:48,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:48,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:48,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:53:48,182 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:53:48,182 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:53:48,182 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:53:48,182 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:53:48,182 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:53:48,183 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:53:48,185 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:48,185 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:53:48,185 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:48,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:48,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:48,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:48,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:48,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:48,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:53:48,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:53:48,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:48,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:48,187 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:48,188 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:48,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:48,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:48,189 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:48,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48240 deadline: 1680062088182, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:48,189 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:48,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48262 deadline: 1680062088183, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:48,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:48,190 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:48,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48198 deadline: 1680062088186, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:48,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:48,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:48,195 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:53:48,195 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:53:48,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:53:48,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:53:48,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:53:48,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:53:48,196 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:48,196 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:48,196 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:48,196 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:48,196 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:48,196 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:53:48,196 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:48,196 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:48,196 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:53:48,201 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:53:48,201 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:53:48,201 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:53:48,201 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:53:48,202 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:48,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48182 deadline: 1680062088189, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:48,207 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=386 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/A/f30a28204ca1442dbb430760c852c3c3 2023-03-29 03:53:48,235 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=386 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/B/df4480d40c66427c971937559b5a7921 2023-03-29 03:53:48,294 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:48,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48262 deadline: 1680062088292, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:48,295 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:48,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48198 deadline: 1680062088292, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:48,296 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:48,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48240 deadline: 1680062088296, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:48,301 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:53:48,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:53:48,305 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:53:48,306 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:48,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48182 deadline: 1680062088304, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:48,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:53:48,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:53:48,307 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:53:48,307 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:53:48,308 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:53:48,308 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:53:48,310 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:48,310 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:48,312 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:48,312 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:53:48,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:53:48,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:53:48,312 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:48,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:48,312 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:53:48,313 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:48,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:53:48,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:48,313 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:53:48,313 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:53:48,314 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:53:48,314 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:53:48,314 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:53:48,316 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:53:48,316 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:53:48,316 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:53:48,316 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:48,316 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. started... 2023-03-29 03:53:48,316 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. as already flushing 2023-03-29 03:53:48,317 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:48,317 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:53:48,317 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:53:48,317 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:48,317 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:53:48,317 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:53:48,318 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:48,318 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:48,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:48,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:48,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:48,319 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:48,319 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:53:48,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:48,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:48,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:48,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:48,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:48,322 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:53:48,322 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c18ffaf[Count = 0] remaining members to acquire global barrier 2023-03-29 03:53:48,322 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:53:48,322 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:48,324 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:48,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:48,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:48,324 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:53:48,324 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:53:48,324 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:53:48,326 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:48,326 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:53:48,328 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:53:48,328 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:53:48,328 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:53:48,328 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:48,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:48,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:48,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:48,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:48,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:48,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:48,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:48,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:48,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:48,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:48,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:53:48,331 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:53:48,331 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:53:48,331 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:53:48,331 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:53:48,331 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:53:48,331 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:53:48,334 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:48,334 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:48,334 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:53:48,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:48,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:48,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:48,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:48,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:48,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:53:48,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:53:48,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:48,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:48,335 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:48,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:48,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:48,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:48,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:48,336 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:48,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:48,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:48,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:48,340 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:48,340 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:53:48,340 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:48,340 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:48,340 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:53:48,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:53:48,340 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:53:48,340 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:53:48,340 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:53:48,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:53:48,340 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:48,341 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:53:48,341 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:53:48,341 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:48,341 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:48,341 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:48,341 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:53:48,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:53:48,359 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:48,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48252 deadline: 1680062088357, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:48,393 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/B/7de8aaa894f94c769396739b622c1c4f as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/7de8aaa894f94c769396739b622c1c4f 2023-03-29 03:53:48,395 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/A/af807381d9554386bb43d0f3f86a287c as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/af807381d9554386bb43d0f3f86a287c 2023-03-29 03:53:48,404 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 52c064cf4695189b45c25208ecad32f7/B of 52c064cf4695189b45c25208ecad32f7 into 7de8aaa894f94c769396739b622c1c4f(size=12.2 K), total size for store is 24.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:53:48,405 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 52c064cf4695189b45c25208ecad32f7: 2023-03-29 03:53:48,405 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7., storeName=52c064cf4695189b45c25208ecad32f7/B, priority=13, startTime=1680062027913; duration=0sec 2023-03-29 03:53:48,406 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:53:48,406 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-29 03:53:48,408 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36798 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-29 03:53:48,408 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1912): 52c064cf4695189b45c25208ecad32f7/C is initiating minor compaction (all files) 2023-03-29 03:53:48,408 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 52c064cf4695189b45c25208ecad32f7/A of 52c064cf4695189b45c25208ecad32f7 into af807381d9554386bb43d0f3f86a287c(size=12.2 K), total size for store is 24.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:53:48,408 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 52c064cf4695189b45c25208ecad32f7/C in TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:48,408 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 52c064cf4695189b45c25208ecad32f7: 2023-03-29 03:53:48,408 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/a2d935a73df840608cacf3c3c925d769, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/d62336621bae428683f69cf215e02184, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/d116f3bf259446dfa2c77b7b983364e3] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp, totalSize=35.9 K 2023-03-29 03:53:48,408 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7., storeName=52c064cf4695189b45c25208ecad32f7/A, priority=13, startTime=1680062027913; duration=0sec 2023-03-29 03:53:48,408 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:53:48,409 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting a2d935a73df840608cacf3c3c925d769, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=332, earliestPutTs=1680062026043 2023-03-29 03:53:48,410 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting d62336621bae428683f69cf215e02184, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=349, earliestPutTs=1680062027235 2023-03-29 03:53:48,411 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting d116f3bf259446dfa2c77b7b983364e3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=368, earliestPutTs=1680062027378 2023-03-29 03:53:48,428 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] throttle.PressureAwareThroughputController(145): 52c064cf4695189b45c25208ecad32f7#C#compaction#35 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:53:48,441 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:53:48,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:53:48,445 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:53:48,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:53:48,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:53:48,447 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:53:48,447 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:53:48,447 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:53:48,447 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:53:48,448 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:48,448 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:48,450 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:53:48,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:53:48,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:53:48,450 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:48,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:48,450 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:48,450 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:53:48,451 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:48,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:53:48,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:48,451 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:53:48,452 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:53:48,452 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:53:48,452 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:53:48,452 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:53:48,452 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:53:48,452 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:53:48,452 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:53:48,452 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:48,452 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. started... 2023-03-29 03:53:48,452 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. as already flushing 2023-03-29 03:53:48,452 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:48,452 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:53:48,452 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:53:48,452 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:48,453 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:53:48,453 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:53:48,456 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:48,456 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:48,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:48,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:48,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:48,456 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:48,456 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:53:48,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:48,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:48,457 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/C/65a9c465022245f9927e6fdbd30bad53 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/65a9c465022245f9927e6fdbd30bad53 2023-03-29 03:53:48,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:48,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:48,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:48,458 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:53:48,458 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@783f4cda[Count = 0] remaining members to acquire global barrier 2023-03-29 03:53:48,458 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:53:48,458 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:48,460 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:48,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:48,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:48,460 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:53:48,460 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:53:48,460 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:53:48,461 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:48,461 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:53:48,462 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:48,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:48,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:48,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:48,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:48,462 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:53:48,463 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:53:48,463 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:53:48,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:48,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:48,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:48,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:48,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:48,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:48,468 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 52c064cf4695189b45c25208ecad32f7/C of 52c064cf4695189b45c25208ecad32f7 into 65a9c465022245f9927e6fdbd30bad53(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:53:48,468 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 52c064cf4695189b45c25208ecad32f7: 2023-03-29 03:53:48,468 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7., storeName=52c064cf4695189b45c25208ecad32f7/C, priority=13, startTime=1680062027913; duration=0sec 2023-03-29 03:53:48,468 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:53:48,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:53:48,471 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:53:48,471 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:53:48,471 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:53:48,471 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:53:48,471 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:53:48,471 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:53:48,473 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:48,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:48,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:48,473 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:48,473 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:53:48,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:48,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:48,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:48,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:48,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:53:48,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:53:48,474 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:48,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:48,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:48,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:48,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:48,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:48,477 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:48,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:48,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:48,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:48,486 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:48,486 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:48,486 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:48,486 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:48,486 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:53:48,486 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:53:48,486 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:53:48,486 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:53:48,486 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:48,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:53:48,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:53:48,487 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:48,487 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:53:48,487 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:53:48,487 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:53:48,487 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:48,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:53:48,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:53:48,499 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:48,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48262 deadline: 1680062088496, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:48,500 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:48,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48198 deadline: 1680062088497, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:48,500 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:48,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48240 deadline: 1680062088497, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:48,510 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:48,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48182 deadline: 1680062088509, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:48,587 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:53:48,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:53:48,591 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:53:48,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:53:48,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:53:48,595 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:53:48,595 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:53:48,595 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:53:48,595 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:53:48,595 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:48,596 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:48,600 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:48,600 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:53:48,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:53:48,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:53:48,600 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:48,601 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:53:48,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:48,601 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:48,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:53:48,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:48,602 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:53:48,602 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:53:48,602 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:53:48,602 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:53:48,602 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:53:48,602 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:53:48,604 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:53:48,604 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:53:48,608 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:48,608 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. started... 2023-03-29 03:53:48,608 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. as already flushing 2023-03-29 03:53:48,608 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:48,608 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:53:48,608 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:53:48,608 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:48,608 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:53:48,608 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:53:48,610 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:48,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:48,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:48,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:48,611 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:48,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:48,612 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:48,612 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:53:48,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:48,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:48,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:48,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:48,615 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:53:48,615 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@691006c8[Count = 0] remaining members to acquire global barrier 2023-03-29 03:53:48,615 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:53:48,616 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:48,618 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:48,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:48,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:48,618 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:53:48,618 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:53:48,618 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:53:48,618 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:48,618 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:53:48,620 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:53:48,620 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:48,620 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:53:48,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:48,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:48,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:48,620 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:53:48,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:48,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:48,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:48,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:48,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:48,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:48,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:48,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:53:48,626 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:53:48,626 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:53:48,626 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:53:48,626 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:53:48,626 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:53:48,626 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:53:48,628 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:48,628 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:48,628 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:53:48,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:48,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:48,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:48,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:48,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:48,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:53:48,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:53:48,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:48,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:48,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:48,630 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:48,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:48,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:48,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:48,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:48,633 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:48,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:48,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:48,638 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:53:48,639 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:53:48,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:53:48,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:53:48,639 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:48,639 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:48,639 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:48,639 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:48,639 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:48,639 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:53:48,639 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:53:48,639 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:53:48,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:53:48,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:53:48,639 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:48,640 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:48,640 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:53:48,640 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:53:48,655 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=386 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/C/c236104cd258408eb3dd68684461d96c 2023-03-29 03:53:48,664 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:48,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48252 deadline: 1680062088663, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:48,666 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/A/f30a28204ca1442dbb430760c852c3c3 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/f30a28204ca1442dbb430760c852c3c3 2023-03-29 03:53:48,678 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/f30a28204ca1442dbb430760c852c3c3, entries=150, sequenceid=386, filesize=11.9 K 2023-03-29 03:53:48,680 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/B/df4480d40c66427c971937559b5a7921 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/df4480d40c66427c971937559b5a7921 2023-03-29 03:53:48,690 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/df4480d40c66427c971937559b5a7921, entries=100, sequenceid=386, filesize=9.5 K 2023-03-29 03:53:48,692 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/C/c236104cd258408eb3dd68684461d96c as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/c236104cd258408eb3dd68684461d96c 2023-03-29 03:53:48,712 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/c236104cd258408eb3dd68684461d96c, entries=100, sequenceid=386, filesize=9.5 K 2023-03-29 03:53:48,713 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 52c064cf4695189b45c25208ecad32f7 in 549ms, sequenceid=386, compaction requested=true 2023-03-29 03:53:48,713 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 52c064cf4695189b45c25208ecad32f7: 2023-03-29 03:53:48,714 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:53:48,714 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-29 03:53:48,714 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:53:48,714 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-29 03:53:48,714 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-29 03:53:48,715 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36862 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-29 03:53:48,715 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1912): 52c064cf4695189b45c25208ecad32f7/A is initiating minor compaction (all files) 2023-03-29 03:53:48,716 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 52c064cf4695189b45c25208ecad32f7/A in TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:48,716 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/af807381d9554386bb43d0f3f86a287c, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/2acbca75782d454daf0cffd810199e73, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/f30a28204ca1442dbb430760c852c3c3] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp, totalSize=36.0 K 2023-03-29 03:53:48,716 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34422 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-29 03:53:48,716 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1912): 52c064cf4695189b45c25208ecad32f7/B is initiating minor compaction (all files) 2023-03-29 03:53:48,716 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 52c064cf4695189b45c25208ecad32f7/B in TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:48,716 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/7de8aaa894f94c769396739b622c1c4f, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/67e255c7e47d4eeeaac0edda435f2968, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/df4480d40c66427c971937559b5a7921] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp, totalSize=33.6 K 2023-03-29 03:53:48,716 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting af807381d9554386bb43d0f3f86a287c, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=349, earliestPutTs=1680062027235 2023-03-29 03:53:48,718 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 7de8aaa894f94c769396739b622c1c4f, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=349, earliestPutTs=1680062027235 2023-03-29 03:53:48,719 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 2acbca75782d454daf0cffd810199e73, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=368, earliestPutTs=1680062027378 2023-03-29 03:53:48,719 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 67e255c7e47d4eeeaac0edda435f2968, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=368, earliestPutTs=1680062027378 2023-03-29 03:53:48,719 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting f30a28204ca1442dbb430760c852c3c3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=386, earliestPutTs=1680062028036 2023-03-29 03:53:48,720 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting df4480d40c66427c971937559b5a7921, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=386, earliestPutTs=1680062028036 2023-03-29 03:53:48,733 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] throttle.PressureAwareThroughputController(145): 52c064cf4695189b45c25208ecad32f7#A#compaction#36 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:53:48,733 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] throttle.PressureAwareThroughputController(145): 52c064cf4695189b45c25208ecad32f7#B#compaction#37 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:53:48,740 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:53:48,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:53:48,745 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:53:48,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:53:48,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:53:48,748 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:53:48,748 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:53:48,748 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:53:48,748 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:53:48,748 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:48,749 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:48,750 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/B/9e329bd513ce4be7ba4d19a1993c0b55 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/9e329bd513ce4be7ba4d19a1993c0b55 2023-03-29 03:53:48,750 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:48,750 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:53:48,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:53:48,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:53:48,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:48,751 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:48,751 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:48,751 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:53:48,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:53:48,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:48,752 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:53:48,757 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:53:48,757 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:53:48,758 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:53:48,758 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:53:48,758 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:53:48,760 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 52c064cf4695189b45c25208ecad32f7/B of 52c064cf4695189b45c25208ecad32f7 into 9e329bd513ce4be7ba4d19a1993c0b55(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:53:48,760 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 52c064cf4695189b45c25208ecad32f7: 2023-03-29 03:53:48,760 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7., storeName=52c064cf4695189b45c25208ecad32f7/B, priority=13, startTime=1680062028714; duration=0sec 2023-03-29 03:53:48,760 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:53:48,760 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-03-29 03:53:48,764 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:53:48,764 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:53:48,764 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:48,764 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. started... 2023-03-29 03:53:48,764 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 52c064cf4695189b45c25208ecad32f7 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-03-29 03:53:48,765 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 52c064cf4695189b45c25208ecad32f7, store=A 2023-03-29 03:53:48,765 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:53:48,765 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 52c064cf4695189b45c25208ecad32f7, store=B 2023-03-29 03:53:48,765 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:53:48,765 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 52c064cf4695189b45c25208ecad32f7, store=C 2023-03-29 03:53:48,765 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:53:48,765 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-03-29 03:53:48,765 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-03-29 03:53:48,766 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. because compaction request was cancelled 2023-03-29 03:53:48,777 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/A/70231a037c60467fb531beed1bd2a65e as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/70231a037c60467fb531beed1bd2a65e 2023-03-29 03:53:48,783 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=404 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/A/d2643fb5e294487eac330c79280e510a 2023-03-29 03:53:48,787 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 52c064cf4695189b45c25208ecad32f7/A of 52c064cf4695189b45c25208ecad32f7 into 70231a037c60467fb531beed1bd2a65e(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:53:48,787 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 52c064cf4695189b45c25208ecad32f7: 2023-03-29 03:53:48,787 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7., storeName=52c064cf4695189b45c25208ecad32f7/A, priority=13, startTime=1680062028714; duration=0sec 2023-03-29 03:53:48,787 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:53:48,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on 52c064cf4695189b45c25208ecad32f7 2023-03-29 03:53:48,805 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. as already flushing 2023-03-29 03:53:48,822 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=404 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/B/a9bd546764ea411d89687fb54dac8f41 2023-03-29 03:53:48,838 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:48,838 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:48,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48182 deadline: 1680062088834, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:48,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48240 deadline: 1680062088835, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:48,838 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:48,838 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:48,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48198 deadline: 1680062088836, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:48,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48262 deadline: 1680062088836, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:48,844 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=404 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/C/ed5ecd8c79904eff96a4ccd90bc6ca11 2023-03-29 03:53:48,852 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/A/d2643fb5e294487eac330c79280e510a as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/d2643fb5e294487eac330c79280e510a 2023-03-29 03:53:48,859 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/d2643fb5e294487eac330c79280e510a, entries=150, sequenceid=404, filesize=11.9 K 2023-03-29 03:53:48,860 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/B/a9bd546764ea411d89687fb54dac8f41 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/a9bd546764ea411d89687fb54dac8f41 2023-03-29 03:53:48,866 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/a9bd546764ea411d89687fb54dac8f41, entries=150, sequenceid=404, filesize=11.9 K 2023-03-29 03:53:48,868 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/C/ed5ecd8c79904eff96a4ccd90bc6ca11 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/ed5ecd8c79904eff96a4ccd90bc6ca11 2023-03-29 03:53:48,884 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/ed5ecd8c79904eff96a4ccd90bc6ca11, entries=150, sequenceid=404, filesize=11.9 K 2023-03-29 03:53:48,889 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=107.34 KB/109920 for 52c064cf4695189b45c25208ecad32f7 in 124ms, sequenceid=404, compaction requested=true 2023-03-29 03:53:48,889 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 52c064cf4695189b45c25208ecad32f7: 2023-03-29 03:53:48,889 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:48,889 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:53:48,890 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:53:48,890 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:48,890 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:53:48,890 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:53:48,892 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:48,892 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:48,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:48,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:48,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:48,893 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:48,893 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:53:48,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:48,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:48,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:48,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:48,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:48,894 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:53:48,894 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@29fe6ed5[Count = 0] remaining members to acquire global barrier 2023-03-29 03:53:48,894 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:53:48,895 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:48,897 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:48,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:48,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:48,897 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:53:48,897 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:53:48,897 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:53:48,897 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:48,897 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:53:48,898 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:53:48,898 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:48,899 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:53:48,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:48,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:48,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:48,899 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:53:48,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:48,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:48,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:48,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:48,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:48,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:48,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:48,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:53:48,902 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:53:48,902 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:53:48,902 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:53:48,902 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:53:48,902 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:53:48,902 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:53:48,904 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:48,904 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:48,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:48,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:48,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:48,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:48,904 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:53:48,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:48,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:48,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:53:48,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:53:48,905 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:48,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:48,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:48,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:48,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:48,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:48,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:48,907 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:48,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:48,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:48,910 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:53:48,910 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:48,910 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:48,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:53:48,910 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:53:48,910 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:53:48,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:53:48,910 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:53:48,910 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:53:48,910 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:48,911 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:48,911 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:48,911 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:48,911 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:48,911 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:53:48,911 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:53:48,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:53:48,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:53:48,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on 52c064cf4695189b45c25208ecad32f7 2023-03-29 03:53:48,942 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 52c064cf4695189b45c25208ecad32f7 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-29 03:53:48,942 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 52c064cf4695189b45c25208ecad32f7, store=A 2023-03-29 03:53:48,942 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:53:48,942 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 52c064cf4695189b45c25208ecad32f7, store=B 2023-03-29 03:53:48,942 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:53:48,942 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 52c064cf4695189b45c25208ecad32f7, store=C 2023-03-29 03:53:48,942 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:53:48,955 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:48,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48262 deadline: 1680062088953, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:48,955 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:48,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48182 deadline: 1680062088954, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:48,959 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:48,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48198 deadline: 1680062088955, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:48,960 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:48,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48240 deadline: 1680062088955, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:48,963 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=425 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/A/e70d5d1fcb4e4814bbcabdfda99e8f81 2023-03-29 03:53:48,989 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=425 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/B/25b4f11b164548afab4e9ec8d417b6cb 2023-03-29 03:53:49,011 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:53:49,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:53:49,015 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:53:49,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:53:49,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:53:49,019 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:53:49,019 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:53:49,019 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:53:49,019 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:53:49,020 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:49,020 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:49,022 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:49,022 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:53:49,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:53:49,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:53:49,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:49,023 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:49,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:53:49,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:49,024 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:53:49,024 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:49,024 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:53:49,024 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:53:49,024 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:53:49,024 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:53:49,024 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:53:49,024 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:53:49,025 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:53:49,025 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:53:49,025 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:49,025 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. started... 2023-03-29 03:53:49,025 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. as already flushing 2023-03-29 03:53:49,025 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:49,025 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:53:49,025 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:53:49,025 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:49,026 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:53:49,026 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:53:49,027 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:49,027 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:49,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:49,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:49,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:49,027 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:49,028 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:53:49,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:49,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:49,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:49,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:49,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:49,029 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:53:49,029 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@229019c2[Count = 0] remaining members to acquire global barrier 2023-03-29 03:53:49,029 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:53:49,029 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:49,031 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:49,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:49,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:49,031 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:53:49,031 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:53:49,031 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:53:49,032 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:49,032 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:53:49,033 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:49,033 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:53:49,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:49,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:49,033 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:53:49,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:49,034 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:53:49,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:49,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:49,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:49,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:49,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:49,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:49,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:49,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:53:49,036 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:53:49,036 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:53:49,036 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:53:49,036 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:53:49,036 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:53:49,037 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:53:49,039 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:49,039 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:49,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:49,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:49,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:49,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:49,040 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:53:49,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:49,040 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:49,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:49,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:53:49,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:53:49,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:49,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:49,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:49,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:49,042 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:49,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:49,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:49,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:49,047 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:49,047 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:49,047 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:49,047 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:49,047 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:49,047 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:49,047 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:49,047 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:53:49,047 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:53:49,048 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:53:49,048 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:53:49,048 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:53:49,048 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:53:49,047 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:53:49,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:53:49,048 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:53:49,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:53:49,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:53:49,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:53:49,057 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:49,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48262 deadline: 1680062089057, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:49,063 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:49,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48198 deadline: 1680062089062, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:49,064 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:49,064 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:49,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48182 deadline: 1680062089062, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:49,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48240 deadline: 1680062089062, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:49,148 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:53:49,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:53:49,152 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:53:49,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:53:49,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:53:49,154 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:53:49,154 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:53:49,154 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:53:49,154 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:53:49,154 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:49,154 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:49,156 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:49,156 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:53:49,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:53:49,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:53:49,157 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:49,157 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:53:49,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:49,157 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:49,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:53:49,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:49,158 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:53:49,159 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:53:49,159 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:53:49,159 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:53:49,159 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:53:49,159 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:53:49,159 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:53:49,159 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:53:49,159 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:49,159 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. started... 2023-03-29 03:53:49,160 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. as already flushing 2023-03-29 03:53:49,160 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:49,160 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:53:49,160 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:53:49,160 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:49,160 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:53:49,160 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:53:49,161 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:49,162 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:49,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:49,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:49,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:49,162 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:49,162 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:53:49,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:49,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:49,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:49,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:49,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:49,165 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:53:49,165 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@204bd8d6[Count = 0] remaining members to acquire global barrier 2023-03-29 03:53:49,165 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:53:49,165 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:49,168 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:49,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:49,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:49,168 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:53:49,168 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:49,168 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:53:49,168 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:53:49,168 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:53:49,170 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:49,170 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:53:49,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:49,170 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:53:49,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:49,170 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:53:49,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:49,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:49,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:49,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:49,171 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:49,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48252 deadline: 1680062089168, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:49,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:49,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:49,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:49,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:49,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:53:49,174 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:53:49,174 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:53:49,174 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:53:49,174 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:53:49,174 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:53:49,174 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:53:49,177 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:49,177 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:53:49,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:49,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:49,177 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:49,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:49,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:49,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:49,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:53:49,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:53:49,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:49,177 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:49,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:49,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:49,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:49,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:49,179 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:49,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:49,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:49,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:49,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:49,181 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:49,181 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:49,181 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:49,181 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:49,181 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:49,181 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:49,181 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:53:49,181 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:53:49,181 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:53:49,181 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:53:49,181 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:49,181 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:53:49,181 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:53:49,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:53:49,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:53:49,182 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:53:49,182 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:53:49,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:53:49,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:53:49,263 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:49,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48262 deadline: 1680062089260, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:49,270 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:49,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48198 deadline: 1680062089266, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:49,270 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:49,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48240 deadline: 1680062089267, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:49,271 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:49,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48182 deadline: 1680062089268, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:49,283 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:53:49,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:53:49,287 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:53:49,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:53:49,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:53:49,290 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:53:49,290 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:53:49,290 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:53:49,290 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:53:49,291 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:49,291 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:49,293 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:53:49,294 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:49,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:53:49,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:53:49,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:49,295 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:49,295 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:49,295 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:53:49,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:53:49,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:49,295 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:53:49,295 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:53:49,296 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:53:49,296 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:53:49,296 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:53:49,296 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:53:49,300 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:53:49,300 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:53:49,300 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:49,300 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. started... 2023-03-29 03:53:49,300 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. as already flushing 2023-03-29 03:53:49,300 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:49,300 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:53:49,300 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:53:49,300 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:49,300 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:53:49,300 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:53:49,302 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:49,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:49,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:49,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:49,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:49,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:49,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:49,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:49,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:49,306 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:53:49,306 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4904ac51[Count = 0] remaining members to acquire global barrier 2023-03-29 03:53:49,306 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:53:49,306 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:49,308 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:49,308 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:53:49,309 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:49,309 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(162): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:49,310 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:49,310 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:53:49,310 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:53:49,310 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:53:49,310 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:53:49,311 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:49,311 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:53:49,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:49,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:49,311 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:53:49,312 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:53:49,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:49,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:49,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:49,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:49,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:49,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:49,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:49,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:49,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:53:49,315 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:53:49,315 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:53:49,315 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:53:49,315 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:53:49,315 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:53:49,315 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:53:49,317 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:49,317 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:49,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:49,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:49,317 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:53:49,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:49,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:49,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:49,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:53:49,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:53:49,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:49,318 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:49,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:49,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:49,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:49,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:49,319 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:49,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:49,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:49,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:49,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:49,323 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:49,323 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:49,323 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:53:49,323 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:49,323 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:49,323 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:49,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:53:49,323 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:49,323 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:49,323 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:49,323 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:53:49,323 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:53:49,323 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:53:49,323 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:53:49,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:53:49,323 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:53:49,324 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:53:49,324 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:53:49,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:53:49,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:53:49,415 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=425 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/C/8362070a6bb944dd9ac0dba88bc28ec7 2023-03-29 03:53:49,423 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/A/e70d5d1fcb4e4814bbcabdfda99e8f81 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/e70d5d1fcb4e4814bbcabdfda99e8f81 2023-03-29 03:53:49,424 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:53:49,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:53:49,428 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:53:49,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:53:49,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:53:49,430 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:53:49,430 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:53:49,430 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:53:49,430 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:53:49,431 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:49,431 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:49,431 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/e70d5d1fcb4e4814bbcabdfda99e8f81, entries=150, sequenceid=425, filesize=11.9 K 2023-03-29 03:53:49,432 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:53:49,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:53:49,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:53:49,432 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:49,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:49,433 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:49,433 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:53:49,433 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:49,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:53:49,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:49,434 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:53:49,434 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/B/25b4f11b164548afab4e9ec8d417b6cb as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/25b4f11b164548afab4e9ec8d417b6cb 2023-03-29 03:53:49,434 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:53:49,434 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:53:49,434 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:53:49,434 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:53:49,434 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:53:49,435 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:53:49,435 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:53:49,435 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:49,435 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. started... 2023-03-29 03:53:49,435 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. as already flushing 2023-03-29 03:53:49,435 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:49,435 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:53:49,435 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:53:49,435 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:49,435 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:53:49,435 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:53:49,437 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:49,437 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:49,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:49,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:49,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:49,438 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:49,438 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:53:49,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:49,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:49,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:49,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:49,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:49,439 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:53:49,440 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@13d6fb62[Count = 0] remaining members to acquire global barrier 2023-03-29 03:53:49,440 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:53:49,440 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:49,441 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/25b4f11b164548afab4e9ec8d417b6cb, entries=150, sequenceid=425, filesize=11.9 K 2023-03-29 03:53:49,442 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:49,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:49,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:49,442 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:53:49,442 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:53:49,442 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:53:49,443 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:49,443 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:53:49,443 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/C/8362070a6bb944dd9ac0dba88bc28ec7 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/8362070a6bb944dd9ac0dba88bc28ec7 2023-03-29 03:53:49,444 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:49,444 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:53:49,444 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:53:49,444 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:53:49,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:49,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:49,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:49,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:49,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:49,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:49,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:49,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:49,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:49,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:49,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:53:49,452 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:53:49,452 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:53:49,452 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:53:49,452 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:53:49,452 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:53:49,452 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:53:49,458 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/8362070a6bb944dd9ac0dba88bc28ec7, entries=150, sequenceid=425, filesize=11.9 K 2023-03-29 03:53:49,460 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:49,460 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:49,460 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:53:49,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:49,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:49,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:49,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:49,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:49,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:49,461 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:49,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:53:49,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:53:49,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:49,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:49,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:49,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:49,463 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=67.09 KB/68700 for 52c064cf4695189b45c25208ecad32f7 in 521ms, sequenceid=425, compaction requested=true 2023-03-29 03:53:49,464 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 52c064cf4695189b45c25208ecad32f7: 2023-03-29 03:53:49,464 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:53:49,464 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:53:49,464 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-29 03:53:49,464 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-29 03:53:49,464 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-29 03:53:49,464 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:49,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:49,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:49,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:49,466 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36968 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-29 03:53:49,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:49,466 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1912): 52c064cf4695189b45c25208ecad32f7/A is initiating minor compaction (all files) 2023-03-29 03:53:49,467 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 52c064cf4695189b45c25208ecad32f7/A in TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:49,467 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/70231a037c60467fb531beed1bd2a65e, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/d2643fb5e294487eac330c79280e510a, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/e70d5d1fcb4e4814bbcabdfda99e8f81] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp, totalSize=36.1 K 2023-03-29 03:53:49,468 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 70231a037c60467fb531beed1bd2a65e, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=386, earliestPutTs=1680062027382 2023-03-29 03:53:49,468 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 46642 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-29 03:53:49,469 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1912): 52c064cf4695189b45c25208ecad32f7/C is initiating minor compaction (all files) 2023-03-29 03:53:49,469 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 52c064cf4695189b45c25208ecad32f7/C in TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:49,469 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/65a9c465022245f9927e6fdbd30bad53, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/c236104cd258408eb3dd68684461d96c, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/ed5ecd8c79904eff96a4ccd90bc6ca11, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/8362070a6bb944dd9ac0dba88bc28ec7] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp, totalSize=45.5 K 2023-03-29 03:53:49,470 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting d2643fb5e294487eac330c79280e510a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=404, earliestPutTs=1680062028180 2023-03-29 03:53:49,470 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 65a9c465022245f9927e6fdbd30bad53, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=368, earliestPutTs=1680062027378 2023-03-29 03:53:49,472 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting e70d5d1fcb4e4814bbcabdfda99e8f81, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=425, earliestPutTs=1680062028826 2023-03-29 03:53:49,473 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting c236104cd258408eb3dd68684461d96c, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=386, earliestPutTs=1680062028036 2023-03-29 03:53:49,473 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting ed5ecd8c79904eff96a4ccd90bc6ca11, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=404, earliestPutTs=1680062028180 2023-03-29 03:53:49,474 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 8362070a6bb944dd9ac0dba88bc28ec7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=425, earliestPutTs=1680062028826 2023-03-29 03:53:49,475 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:49,475 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:49,475 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:53:49,475 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:53:49,475 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:53:49,475 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:53:49,475 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:49,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:53:49,475 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:53:49,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:53:49,475 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:49,475 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:53:49,475 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:49,475 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:53:49,475 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:49,476 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:49,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:53:49,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:53:49,489 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] throttle.PressureAwareThroughputController(145): 52c064cf4695189b45c25208ecad32f7#A#compaction#44 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:53:49,492 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] throttle.PressureAwareThroughputController(145): 52c064cf4695189b45c25208ecad32f7#C#compaction#45 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:53:49,551 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/A/3ae9835a527d43acb057d992b3fc1586 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/3ae9835a527d43acb057d992b3fc1586 2023-03-29 03:53:49,555 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/C/da1dc0a233214ed8bd9a00cf08938c6d as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/da1dc0a233214ed8bd9a00cf08938c6d 2023-03-29 03:53:49,564 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 52c064cf4695189b45c25208ecad32f7/A of 52c064cf4695189b45c25208ecad32f7 into 3ae9835a527d43acb057d992b3fc1586(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:53:49,564 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 52c064cf4695189b45c25208ecad32f7: 2023-03-29 03:53:49,565 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7., storeName=52c064cf4695189b45c25208ecad32f7/A, priority=13, startTime=1680062029464; duration=0sec 2023-03-29 03:53:49,565 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:53:49,565 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-29 03:53:49,568 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36968 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-29 03:53:49,568 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1912): 52c064cf4695189b45c25208ecad32f7/B is initiating minor compaction (all files) 2023-03-29 03:53:49,568 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 52c064cf4695189b45c25208ecad32f7/B in TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:49,568 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/9e329bd513ce4be7ba4d19a1993c0b55, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/a9bd546764ea411d89687fb54dac8f41, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/25b4f11b164548afab4e9ec8d417b6cb] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp, totalSize=36.1 K 2023-03-29 03:53:49,568 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 52c064cf4695189b45c25208ecad32f7/C of 52c064cf4695189b45c25208ecad32f7 into da1dc0a233214ed8bd9a00cf08938c6d(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:53:49,569 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 52c064cf4695189b45c25208ecad32f7: 2023-03-29 03:53:49,569 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7., storeName=52c064cf4695189b45c25208ecad32f7/C, priority=12, startTime=1680062029464; duration=0sec 2023-03-29 03:53:49,569 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:53:49,571 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 9e329bd513ce4be7ba4d19a1993c0b55, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=386, earliestPutTs=1680062027382 2023-03-29 03:53:49,576 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:53:49,572 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 52c064cf4695189b45c25208ecad32f7 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-03-29 03:53:49,577 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 52c064cf4695189b45c25208ecad32f7, store=A 2023-03-29 03:53:49,577 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:53:49,577 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 52c064cf4695189b45c25208ecad32f7, store=B 2023-03-29 03:53:49,578 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:53:49,578 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 52c064cf4695189b45c25208ecad32f7, store=C 2023-03-29 03:53:49,578 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:53:49,578 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting a9bd546764ea411d89687fb54dac8f41, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=404, earliestPutTs=1680062028180 2023-03-29 03:53:49,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on 52c064cf4695189b45c25208ecad32f7 2023-03-29 03:53:49,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:53:49,579 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 25b4f11b164548afab4e9ec8d417b6cb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=425, earliestPutTs=1680062028826 2023-03-29 03:53:49,582 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:53:49,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:53:49,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:53:49,589 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:53:49,589 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:53:49,589 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:53:49,589 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:53:49,590 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:49,590 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:49,591 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:53:49,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:53:49,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:53:49,592 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:49,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:49,592 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:49,592 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:49,593 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:53:49,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:53:49,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:49,593 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:53:49,602 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:53:49,602 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:53:49,602 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:53:49,602 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:53:49,602 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:53:49,604 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] throttle.PressureAwareThroughputController(145): 52c064cf4695189b45c25208ecad32f7#B#compaction#47 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:53:49,608 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:53:49,608 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:53:49,608 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:49,608 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. started... 2023-03-29 03:53:49,608 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. as already flushing 2023-03-29 03:53:49,608 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:49,608 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:53:49,608 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:53:49,609 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:49,611 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:49,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48182 deadline: 1680062089606, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:49,612 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:53:49,612 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:53:49,612 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:49,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48240 deadline: 1680062089607, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:49,614 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:49,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48198 deadline: 1680062089611, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:49,614 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:49,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48262 deadline: 1680062089612, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:49,617 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:49,617 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:49,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:49,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:49,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:49,617 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:49,617 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:53:49,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:49,619 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=441 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/A/1cffb05cc65a45758c1ec05e0230a5c4 2023-03-29 03:53:49,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:49,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:49,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:49,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:49,626 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:53:49,626 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@312abed3[Count = 0] remaining members to acquire global barrier 2023-03-29 03:53:49,626 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:53:49,626 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:49,628 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:49,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:49,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:49,628 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:53:49,628 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:53:49,628 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:53:49,629 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:49,629 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:53:49,630 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:53:49,630 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:53:49,630 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:53:49,630 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:49,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:49,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:49,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:49,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:49,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:49,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:49,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:49,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:49,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:49,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:49,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:53:49,635 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:53:49,635 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:53:49,635 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:53:49,635 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:53:49,635 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:53:49,635 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:53:49,638 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:49,638 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:53:49,638 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:49,638 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/B/b66c44ae8220416d8166d2cbe50a52e8 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/b66c44ae8220416d8166d2cbe50a52e8 2023-03-29 03:53:49,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:49,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:49,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:49,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:49,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:49,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:53:49,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:53:49,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:49,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:49,641 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:49,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:49,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:49,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:49,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:49,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:49,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:49,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:49,647 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:49,652 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:53:49,652 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:53:49,652 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:49,652 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:53:49,652 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:53:49,652 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:53:49,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:53:49,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:53:49,652 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:49,652 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:53:49,653 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:53:49,652 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:49,653 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:49,653 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:49,653 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:49,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:53:49,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:53:49,653 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:49,654 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=441 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/B/96cac26339214e14b2e84fe8f7e73f28 2023-03-29 03:53:49,668 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 52c064cf4695189b45c25208ecad32f7/B of 52c064cf4695189b45c25208ecad32f7 into b66c44ae8220416d8166d2cbe50a52e8(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:53:49,668 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 52c064cf4695189b45c25208ecad32f7: 2023-03-29 03:53:49,668 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7., storeName=52c064cf4695189b45c25208ecad32f7/B, priority=13, startTime=1680062029464; duration=0sec 2023-03-29 03:53:49,668 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:53:49,715 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:49,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48182 deadline: 1680062089713, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:49,715 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:49,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48240 deadline: 1680062089714, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:49,716 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:49,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48198 deadline: 1680062089716, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:49,716 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:49,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48262 deadline: 1680062089716, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:49,753 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:53:49,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:53:49,756 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:53:49,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:53:49,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:53:49,758 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:53:49,758 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:53:49,758 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:53:49,758 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:53:49,758 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:49,758 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:49,759 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:53:49,760 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:49,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:53:49,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:53:49,760 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:49,760 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:53:49,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:49,760 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:49,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:53:49,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:49,761 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:53:49,761 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:53:49,761 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:53:49,761 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:53:49,761 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:53:49,761 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:53:49,762 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:53:49,762 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:53:49,762 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:49,762 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. started... 2023-03-29 03:53:49,762 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. as already flushing 2023-03-29 03:53:49,762 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:49,762 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:53:49,762 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:53:49,762 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:49,762 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:53:49,762 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:53:49,763 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:49,763 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:49,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:49,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:49,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:49,764 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:49,764 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:53:49,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:49,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:49,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:49,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:49,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:49,765 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:53:49,765 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f595d48[Count = 0] remaining members to acquire global barrier 2023-03-29 03:53:49,765 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:53:49,766 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:49,767 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:49,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:49,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:49,767 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:53:49,767 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:53:49,767 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:53:49,767 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:49,767 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:53:49,769 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:49,769 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:53:49,770 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:53:49,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:49,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:49,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:49,770 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:53:49,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:49,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:49,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:49,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:49,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:49,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:49,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:49,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:53:49,772 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:53:49,772 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:53:49,772 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:53:49,772 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:53:49,772 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:53:49,773 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:53:49,774 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:49,774 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:53:49,774 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:49,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:49,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:49,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:49,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:49,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:49,775 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:49,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:53:49,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:53:49,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:49,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:49,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:49,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:49,776 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:49,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:49,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:49,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:49,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:49,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:49,779 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:53:49,779 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:49,779 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:49,779 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:49,779 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:53:49,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:53:49,779 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:53:49,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:53:49,779 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:49,779 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:53:49,779 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:53:49,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:53:49,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:53:49,780 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:53:49,780 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:53:49,780 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:53:49,779 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:49,780 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:49,780 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:49,880 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:53:49,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:53:49,883 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:53:49,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:53:49,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:53:49,885 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:53:49,885 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:53:49,885 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:53:49,885 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:53:49,885 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:49,885 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:49,887 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:49,887 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:53:49,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:53:49,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:53:49,887 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:49,887 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:53:49,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:49,887 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:49,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:53:49,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:49,888 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:53:49,888 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:53:49,888 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:53:49,888 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:53:49,888 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:53:49,888 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:53:49,889 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:53:49,889 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:53:49,889 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:49,889 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. started... 2023-03-29 03:53:49,889 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. as already flushing 2023-03-29 03:53:49,890 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:49,890 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:53:49,890 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:53:49,890 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:49,890 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:53:49,890 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:53:49,892 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:49,892 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:49,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:49,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:49,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:49,892 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:49,893 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:53:49,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:49,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:49,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:49,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:49,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:49,894 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:53:49,894 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@23d306d6[Count = 0] remaining members to acquire global barrier 2023-03-29 03:53:49,894 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:53:49,894 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:49,896 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:49,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:49,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:49,896 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:53:49,896 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:53:49,896 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:53:49,897 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:49,897 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:53:49,898 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:53:49,898 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:49,898 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:53:49,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:49,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:49,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:49,898 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:53:49,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:49,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:49,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:49,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:49,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:49,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:49,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:49,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:53:49,901 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:53:49,901 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:53:49,901 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:53:49,917 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:53:49,917 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:53:49,917 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:53:49,919 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:49,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48198 deadline: 1680062089919, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:49,919 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:49,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48182 deadline: 1680062089919, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:49,920 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:49,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48262 deadline: 1680062089919, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:49,920 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:49,920 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:49,920 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:53:49,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:49,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:49,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:49,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:49,921 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:49,922 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:49,925 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:49,925 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:49,925 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:49,925 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:49,925 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:49,925 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:49,925 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:53:49,919 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:49,926 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:53:49,930 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:53:49,926 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:53:49,925 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:49,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:49,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:49,930 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:53:49,930 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:53:49,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48240 deadline: 1680062089919, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:49,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:49,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:49,931 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:53:49,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:53:49,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:49,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:53:49,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:53:49,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:53:49,932 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:53:50,030 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:53:50,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:53:50,035 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:53:50,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:53:50,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:53:50,048 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:53:50,048 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:53:50,048 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:53:50,048 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:53:50,049 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:50,049 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:50,050 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:53:50,050 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:50,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:53:50,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:53:50,051 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:50,051 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:53:50,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:50,051 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:50,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:53:50,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:50,052 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:53:50,060 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:53:50,060 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:53:50,060 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:53:50,060 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:53:50,063 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:53:50,063 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:53:50,064 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:50,064 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. started... 2023-03-29 03:53:50,064 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. as already flushing 2023-03-29 03:53:50,069 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:50,069 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:53:50,070 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:53:50,070 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:50,070 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:53:50,070 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:53:50,071 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:50,071 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:50,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:50,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:50,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:50,072 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:50,072 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:53:50,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:50,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:50,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:50,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:50,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:50,074 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:53:50,074 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7fae425[Count = 0] remaining members to acquire global barrier 2023-03-29 03:53:50,074 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:53:50,074 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:50,079 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:50,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:50,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:50,079 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:53:50,079 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:53:50,079 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:53:50,079 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:50,079 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:53:50,081 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:50,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:50,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:50,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:50,081 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:53:50,082 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:53:50,082 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:53:50,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:50,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:50,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:50,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:50,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:50,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:50,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:50,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:53:50,085 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:53:50,085 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:53:50,085 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:53:50,085 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:53:50,085 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:53:50,085 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:53:50,086 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=441 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/C/604081a3c79341058b1dc38ce0aa3fa6 2023-03-29 03:53:50,089 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:50,089 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:53:50,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:50,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:50,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:53:50,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:53:50,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:50,092 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:50,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:50,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:50,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:50,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:50,099 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:50,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:50,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:50,102 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/A/1cffb05cc65a45758c1ec05e0230a5c4 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/1cffb05cc65a45758c1ec05e0230a5c4 2023-03-29 03:53:50,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:50,102 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:50,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:50,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:50,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:50,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:50,107 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:53:50,107 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:50,107 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:50,107 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:53:50,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:53:50,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:53:50,107 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:50,107 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:53:50,107 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:53:50,107 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:53:50,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:53:50,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:53:50,108 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:53:50,107 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:50,108 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:50,108 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:50,108 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:50,108 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:53:50,108 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:53:50,111 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/1cffb05cc65a45758c1ec05e0230a5c4, entries=200, sequenceid=441, filesize=14.3 K 2023-03-29 03:53:50,113 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/B/96cac26339214e14b2e84fe8f7e73f28 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/96cac26339214e14b2e84fe8f7e73f28 2023-03-29 03:53:50,120 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/96cac26339214e14b2e84fe8f7e73f28, entries=150, sequenceid=441, filesize=11.9 K 2023-03-29 03:53:50,121 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/C/604081a3c79341058b1dc38ce0aa3fa6 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/604081a3c79341058b1dc38ce0aa3fa6 2023-03-29 03:53:50,129 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/604081a3c79341058b1dc38ce0aa3fa6, entries=150, sequenceid=441, filesize=11.9 K 2023-03-29 03:53:50,130 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 52c064cf4695189b45c25208ecad32f7 in 558ms, sequenceid=441, compaction requested=false 2023-03-29 03:53:50,131 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 52c064cf4695189b45c25208ecad32f7: 2023-03-29 03:53:50,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on 52c064cf4695189b45c25208ecad32f7 2023-03-29 03:53:50,197 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 52c064cf4695189b45c25208ecad32f7 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-29 03:53:50,197 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 52c064cf4695189b45c25208ecad32f7, store=A 2023-03-29 03:53:50,197 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:53:50,197 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 52c064cf4695189b45c25208ecad32f7, store=B 2023-03-29 03:53:50,197 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:53:50,197 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 52c064cf4695189b45c25208ecad32f7, store=C 2023-03-29 03:53:50,197 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:53:50,208 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:53:50,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:53:50,213 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:53:50,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:53:50,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:53:50,223 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:53:50,223 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:53:50,223 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:53:50,223 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:53:50,223 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:50,223 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:50,225 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:50,225 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:53:50,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:53:50,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:53:50,225 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:50,225 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:53:50,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:50,226 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:50,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:53:50,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:50,226 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:53:50,234 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:53:50,234 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:53:50,234 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:53:50,235 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:53:50,235 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:53:50,241 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:53:50,241 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:50,242 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. started... 2023-03-29 03:53:50,241 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:53:50,242 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. as already flushing 2023-03-29 03:53:50,242 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:50,242 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:53:50,242 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:53:50,242 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:50,242 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:53:50,242 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:53:50,244 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:50,244 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:50,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:50,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:50,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:50,244 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:50,244 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:53:50,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:50,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:50,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:50,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:50,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:50,245 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:53:50,246 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@63421e2f[Count = 0] remaining members to acquire global barrier 2023-03-29 03:53:50,246 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:53:50,246 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:50,247 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:50,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:50,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:50,247 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:53:50,247 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:53:50,247 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:53:50,247 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:50,247 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:53:50,249 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:53:50,249 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:50,249 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:53:50,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:50,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:50,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:50,250 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:53:50,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:50,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:50,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:50,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:50,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:50,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:50,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:50,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:53:50,252 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:53:50,252 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:53:50,252 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:53:50,252 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:53:50,252 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:53:50,252 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:53:50,254 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:50,254 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:50,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:50,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:50,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:50,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:50,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:50,254 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:53:50,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:53:50,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:53:50,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:50,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:50,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:50,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:50,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:50,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:50,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:50,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:50,258 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:50,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:50,259 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:50,263 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:50,263 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:50,263 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:53:50,263 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:50,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48262 deadline: 1680062090256, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:50,263 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:50,263 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:53:50,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:53:50,263 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:50,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:53:50,264 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:53:50,264 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:53:50,264 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:53:50,264 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:50,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48182 deadline: 1680062090257, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:50,264 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:53:50,264 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:53:50,264 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:50,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:53:50,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:53:50,264 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:50,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48252 deadline: 1680062090260, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:50,264 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:50,265 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:50,268 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:50,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48240 deadline: 1680062090262, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:50,269 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:50,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48198 deadline: 1680062090263, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:50,364 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:53:50,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:53:50,368 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:53:50,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:53:50,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:53:50,369 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:53:50,369 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:53:50,369 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:53:50,369 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:53:50,370 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:50,370 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:50,370 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:50,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48262 deadline: 1680062090365, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:50,370 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:50,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48252 deadline: 1680062090366, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:50,371 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:50,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48182 deadline: 1680062090366, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:50,371 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:50,371 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:53:50,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48240 deadline: 1680062090369, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:50,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:53:50,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:53:50,371 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:50,372 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:50,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48198 deadline: 1680062090370, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:50,372 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:50,372 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:53:50,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:50,373 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:50,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:53:50,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:50,373 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:53:50,373 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:53:50,373 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:53:50,373 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:53:50,373 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:53:50,373 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:53:50,374 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:53:50,374 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:53:50,374 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:50,374 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. started... 2023-03-29 03:53:50,374 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. as already flushing 2023-03-29 03:53:50,374 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:50,374 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:53:50,374 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:53:50,374 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:50,374 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:53:50,374 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:53:50,377 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:50,377 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:50,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:50,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:50,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:50,377 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:50,377 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:53:50,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:50,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:50,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:50,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:50,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:50,384 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:53:50,384 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@66f4bc12[Count = 0] remaining members to acquire global barrier 2023-03-29 03:53:50,384 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:53:50,385 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:50,386 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:50,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:50,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:50,386 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:53:50,386 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:53:50,386 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:53:50,387 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:50,387 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:53:50,389 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:50,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:50,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:50,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:50,389 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:53:50,389 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:53:50,389 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:53:50,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:50,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:50,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:50,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:50,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:50,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:50,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:50,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:53:50,392 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:53:50,392 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:53:50,392 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:53:50,392 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:53:50,392 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:53:50,392 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:53:50,394 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:50,394 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:50,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:50,394 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:53:50,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:50,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:50,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:50,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:50,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:53:50,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:53:50,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:50,399 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:50,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:50,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:50,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:50,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:50,401 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:50,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:50,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:50,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:50,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:50,405 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:53:50,405 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:50,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:53:50,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:53:50,405 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:50,405 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:50,405 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:50,405 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:50,405 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:50,405 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:50,405 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:53:50,405 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:53:50,405 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:53:50,405 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:53:50,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:53:50,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:53:50,406 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:53:50,406 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:53:50,405 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:53:50,506 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:53:50,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:53:50,511 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:53:50,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:53:50,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:53:50,513 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:53:50,513 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:53:50,513 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:53:50,513 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:53:50,514 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:50,514 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:50,515 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:53:50,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:53:50,516 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:50,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:53:50,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:50,517 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:50,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:53:50,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:50,517 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:50,517 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:53:50,517 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:53:50,518 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:53:50,518 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:53:50,518 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:53:50,518 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:53:50,518 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:53:50,518 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:53:50,518 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:53:50,518 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:50,518 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. started... 2023-03-29 03:53:50,518 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. as already flushing 2023-03-29 03:53:50,518 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:50,518 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:53:50,518 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:53:50,518 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:50,519 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:53:50,519 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:53:50,520 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:50,520 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:50,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:50,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:50,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:50,521 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:50,521 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:53:50,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:50,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:50,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:50,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:50,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:50,523 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:53:50,523 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5daae3ac[Count = 0] remaining members to acquire global barrier 2023-03-29 03:53:50,523 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:53:50,523 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:50,524 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:50,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:50,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:50,524 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:53:50,524 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:53:50,524 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:53:50,525 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:50,525 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:53:50,529 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:53:50,529 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:53:50,529 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:53:50,529 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:50,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:50,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:50,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:50,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:50,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:50,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:50,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:50,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:50,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:50,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:50,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:53:50,533 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:53:50,533 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:53:50,533 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:53:50,533 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:53:50,533 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:53:50,533 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:53:50,535 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:50,535 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:53:50,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:50,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:50,535 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:50,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:53:50,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:53:50,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:50,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:50,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:50,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:50,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:50,538 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:50,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:50,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:50,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:50,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:50,540 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:50,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:50,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:50,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:50,542 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:53:50,542 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:53:50,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:53:50,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:53:50,542 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:50,543 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:50,543 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:50,543 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:50,543 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:50,543 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:53:50,543 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:53:50,543 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:53:50,543 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:50,543 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:50,543 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:53:50,543 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:53:50,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:53:50,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:53:50,573 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:50,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:48252 deadline: 1680062090572, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:50,573 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:50,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48182 deadline: 1680062090573, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:50,574 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:50,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48262 deadline: 1680062090573, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:50,575 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:50,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48198 deadline: 1680062090573, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:50,575 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:50,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48240 deadline: 1680062090574, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:50,643 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:53:50,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:53:50,646 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=461 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/A/a2cde79c7f864bfb9e3751080a59bd22 2023-03-29 03:53:50,649 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:53:50,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:53:50,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:53:50,657 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:53:50,657 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:53:50,657 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:53:50,657 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:53:50,657 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:50,657 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:50,659 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:53:50,659 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:53:50,659 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:50,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:53:50,659 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:50,659 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:53:50,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:50,660 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:50,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:53:50,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:50,661 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:53:50,672 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:53:50,672 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:53:50,672 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:53:50,672 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:53:50,672 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:53:50,679 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:53:50,679 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:50,679 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:53:50,679 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. started... 2023-03-29 03:53:50,679 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. as already flushing 2023-03-29 03:53:50,679 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:50,680 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:53:50,680 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:53:50,680 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:50,685 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:53:50,685 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:53:50,695 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:50,695 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:50,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:50,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:50,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:50,695 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:50,695 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:53:50,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:50,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:50,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:50,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:50,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:50,701 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:53:50,701 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@da1eff0[Count = 0] remaining members to acquire global barrier 2023-03-29 03:53:50,701 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:53:50,701 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:50,703 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:50,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:50,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:50,703 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:53:50,703 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:53:50,703 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:53:50,704 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:50,704 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:53:50,707 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:53:50,707 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:53:50,707 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:53:50,707 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:50,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:50,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:50,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:50,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:50,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:50,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:50,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:50,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:50,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:50,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:50,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:53:50,711 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:53:50,711 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:53:50,711 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:53:50,711 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:53:50,711 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:53:50,711 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:53:50,715 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:50,715 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:53:50,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:50,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:50,715 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:50,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:53:50,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:53:50,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:50,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:50,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:50,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:50,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:50,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:50,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:50,721 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:50,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:50,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:50,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:50,726 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:50,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:50,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:50,732 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:53:50,732 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:53:50,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:53:50,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:53:50,733 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:50,733 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:50,733 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:50,734 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:50,734 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:50,734 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:50,734 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:53:50,734 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:53:50,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:53:50,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:53:50,734 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:53:50,734 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:53:50,734 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:53:50,734 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:50,834 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:53:50,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:53:50,838 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:53:50,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:53:50,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:53:50,839 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:53:50,839 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:53:50,839 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:53:50,839 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:53:50,840 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:50,840 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:50,841 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:53:50,841 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:50,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:53:50,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:53:50,842 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:50,842 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:53:50,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:50,842 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:50,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:53:50,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:50,843 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:53:50,843 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:53:50,843 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:53:50,843 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:53:50,843 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:53:50,843 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:53:50,844 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:53:50,844 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:53:50,844 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:50,844 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. started... 2023-03-29 03:53:50,844 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. as already flushing 2023-03-29 03:53:50,844 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:50,844 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:53:50,844 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:53:50,844 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:50,844 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:53:50,844 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:53:50,846 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:50,847 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:50,847 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:53:50,847 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:50,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:50,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:50,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:50,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:50,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:50,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:50,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:50,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:50,849 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:53:50,849 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7e613b0d[Count = 0] remaining members to acquire global barrier 2023-03-29 03:53:50,849 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:53:50,849 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:50,851 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:50,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:50,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:50,851 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:53:50,852 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:53:50,852 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:53:50,852 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:50,852 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:53:50,853 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:50,853 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:53:50,853 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:53:50,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:50,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:50,853 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:53:50,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:50,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:50,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:50,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:50,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:50,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:50,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:50,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:50,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:53:50,857 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:53:50,857 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:53:50,857 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:53:50,857 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:53:50,857 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:53:50,857 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:53:50,859 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:50,859 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:53:50,859 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:50,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:50,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:50,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:50,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:50,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:50,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:53:50,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:53:50,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:50,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:50,860 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:50,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:50,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:50,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:50,863 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:50,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:50,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:50,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:50,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:50,870 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:50,870 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:50,870 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:50,870 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:50,870 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:50,871 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:50,871 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:50,871 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:53:50,871 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:53:50,871 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:53:50,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:53:50,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:53:50,871 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:53:50,871 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:53:50,871 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:53:50,871 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:53:50,872 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:53:50,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:53:50,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:53:50,875 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:50,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48252 deadline: 1680062090875, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:50,879 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:50,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48182 deadline: 1680062090876, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:50,883 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:50,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48198 deadline: 1680062090877, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:50,883 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:50,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48262 deadline: 1680062090878, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:50,884 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:50,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48240 deadline: 1680062090879, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:50,972 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:53:50,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:53:50,976 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:53:50,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:53:50,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:53:50,978 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:53:50,978 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:53:50,978 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:53:50,978 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:53:50,979 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:50,979 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:50,981 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:53:50,981 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:50,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:53:50,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:53:50,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:50,982 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:50,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:53:50,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:50,982 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:53:50,982 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:53:50,982 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:53:50,982 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:53:50,983 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:53:50,983 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:53:50,983 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:53:50,983 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:53:50,983 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:50,983 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. started... 2023-03-29 03:53:50,983 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. as already flushing 2023-03-29 03:53:50,983 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:50,983 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:53:50,983 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:53:50,983 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:50,983 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:53:50,983 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:53:50,985 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:50,985 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:50,985 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:53:50,985 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@74f1261c[Count = 0] remaining members to acquire global barrier 2023-03-29 03:53:50,985 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:53:50,985 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:53:50,985 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:50,986 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:50,986 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:53:50,986 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:50,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:50,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:50,987 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:53:50,987 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:53:50,987 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:53:50,987 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:50,987 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:53:50,989 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:53:50,989 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:53:50,989 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:53:50,989 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:50,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:50,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:50,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:50,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:50,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:50,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:50,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:50,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:50,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:50,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:50,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:53:50,991 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:53:50,991 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:53:50,992 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:53:50,992 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:53:50,992 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:53:50,992 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:53:50,994 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:50,994 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:50,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:50,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:50,994 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:53:50,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:50,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:50,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:50,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:53:50,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:53:50,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:50,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:50,995 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:50,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:50,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:50,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:50,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:50,997 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:50,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:50,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:50,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:51,001 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:51,001 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:51,001 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:53:51,001 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:53:51,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:53:51,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:53:51,001 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:51,001 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:51,002 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:51,002 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:51,002 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:51,001 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:53:51,002 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:53:51,001 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:53:51,002 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:53:51,002 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:53:51,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:53:51,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:53:51,102 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:53:51,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:53:51,106 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:53:51,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:53:51,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:53:51,107 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:53:51,107 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:53:51,107 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:53:51,107 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:53:51,108 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:51,108 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:51,113 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:51,113 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:53:51,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:53:51,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:53:51,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:51,115 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:51,115 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:53:51,115 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:51,115 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=461 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/B/7a54d5f889e9430fa4f451ce6685920e 2023-03-29 03:53:51,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:53:51,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:51,115 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:53:51,116 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:53:51,116 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:53:51,116 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:53:51,116 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:53:51,116 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:53:51,116 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:53:51,116 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:53:51,116 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:51,116 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. started... 2023-03-29 03:53:51,116 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. as already flushing 2023-03-29 03:53:51,116 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:51,117 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:53:51,117 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:53:51,117 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:51,117 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:53:51,117 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:53:51,120 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:51,120 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:51,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:51,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:51,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:51,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:51,121 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:51,121 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:53:51,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:51,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:51,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:51,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:51,125 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:53:51,125 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3db60cff[Count = 0] remaining members to acquire global barrier 2023-03-29 03:53:51,125 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:53:51,125 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:51,126 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:51,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:51,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:51,127 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:53:51,127 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:53:51,127 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:53:51,128 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:51,128 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:53:51,128 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:53:51,129 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:53:51,129 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:53:51,130 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:51,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:51,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:51,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:51,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:51,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:51,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:51,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:51,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:51,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:51,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:51,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:53:51,134 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:53:51,134 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:53:51,134 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:53:51,134 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:53:51,134 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:53:51,134 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:53:51,137 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:51,137 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:51,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:51,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:51,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:51,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:51,137 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:53:51,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:51,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:53:51,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:53:51,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:51,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:51,138 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:51,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:51,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:51,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:51,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:51,139 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:51,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:51,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:51,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:51,142 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:51,142 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:51,142 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:51,142 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:51,142 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:51,142 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:51,142 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:51,142 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:53:51,142 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:53:51,142 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:53:51,143 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:53:51,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:53:51,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:53:51,143 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:53:51,143 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:53:51,142 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:53:51,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:53:51,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:53:51,155 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=461 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/C/8bc409548e5a42f6895639d98eb01278 2023-03-29 03:53:51,164 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/A/a2cde79c7f864bfb9e3751080a59bd22 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/a2cde79c7f864bfb9e3751080a59bd22 2023-03-29 03:53:51,172 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/a2cde79c7f864bfb9e3751080a59bd22, entries=150, sequenceid=461, filesize=11.9 K 2023-03-29 03:53:51,173 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/B/7a54d5f889e9430fa4f451ce6685920e as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/7a54d5f889e9430fa4f451ce6685920e 2023-03-29 03:53:51,181 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/7a54d5f889e9430fa4f451ce6685920e, entries=150, sequenceid=461, filesize=11.9 K 2023-03-29 03:53:51,184 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/C/8bc409548e5a42f6895639d98eb01278 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/8bc409548e5a42f6895639d98eb01278 2023-03-29 03:53:51,190 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/8bc409548e5a42f6895639d98eb01278, entries=150, sequenceid=461, filesize=11.9 K 2023-03-29 03:53:51,191 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 52c064cf4695189b45c25208ecad32f7 in 994ms, sequenceid=461, compaction requested=true 2023-03-29 03:53:51,191 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 52c064cf4695189b45c25208ecad32f7: 2023-03-29 03:53:51,191 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:53:51,192 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:53:51,192 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-29 03:53:51,192 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-29 03:53:51,192 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:53:51,193 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39510 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-29 03:53:51,193 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1912): 52c064cf4695189b45c25208ecad32f7/A is initiating minor compaction (all files) 2023-03-29 03:53:51,193 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37070 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-29 03:53:51,193 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 52c064cf4695189b45c25208ecad32f7/A in TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:51,193 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1912): 52c064cf4695189b45c25208ecad32f7/B is initiating minor compaction (all files) 2023-03-29 03:53:51,194 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/3ae9835a527d43acb057d992b3fc1586, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/1cffb05cc65a45758c1ec05e0230a5c4, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/a2cde79c7f864bfb9e3751080a59bd22] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp, totalSize=38.6 K 2023-03-29 03:53:51,194 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 52c064cf4695189b45c25208ecad32f7/B in TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:51,194 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/b66c44ae8220416d8166d2cbe50a52e8, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/96cac26339214e14b2e84fe8f7e73f28, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/7a54d5f889e9430fa4f451ce6685920e] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp, totalSize=36.2 K 2023-03-29 03:53:51,194 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 3ae9835a527d43acb057d992b3fc1586, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=425, earliestPutTs=1680062028826 2023-03-29 03:53:51,194 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting b66c44ae8220416d8166d2cbe50a52e8, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=425, earliestPutTs=1680062028826 2023-03-29 03:53:51,202 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 1cffb05cc65a45758c1ec05e0230a5c4, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=441, earliestPutTs=1680062028944 2023-03-29 03:53:51,202 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 96cac26339214e14b2e84fe8f7e73f28, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=441, earliestPutTs=1680062028944 2023-03-29 03:53:51,202 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting a2cde79c7f864bfb9e3751080a59bd22, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=461, earliestPutTs=1680062029597 2023-03-29 03:53:51,203 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 7a54d5f889e9430fa4f451ce6685920e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=461, earliestPutTs=1680062029597 2023-03-29 03:53:51,218 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] throttle.PressureAwareThroughputController(145): 52c064cf4695189b45c25208ecad32f7#B#compaction#53 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:53:51,226 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] throttle.PressureAwareThroughputController(145): 52c064cf4695189b45c25208ecad32f7#A#compaction#54 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:53:51,243 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:53:51,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:53:51,248 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:53:51,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:53:51,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:53:51,256 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:53:51,256 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:53:51,256 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:53:51,256 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:53:51,257 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:51,257 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:51,258 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:53:51,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:53:51,259 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:51,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:53:51,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:51,260 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:51,260 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:53:51,260 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:51,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:53:51,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:51,261 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:53:51,266 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/A/a61dc4c8d4f2402db89d0911daeef838 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/a61dc4c8d4f2402db89d0911daeef838 2023-03-29 03:53:51,266 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/B/1f19019b749a41ed95fd3f201f6e510e as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/1f19019b749a41ed95fd3f201f6e510e 2023-03-29 03:53:51,271 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:53:51,271 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:53:51,271 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:53:51,272 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:53:51,272 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:53:51,281 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 52c064cf4695189b45c25208ecad32f7/A of 52c064cf4695189b45c25208ecad32f7 into a61dc4c8d4f2402db89d0911daeef838(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:53:51,281 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 52c064cf4695189b45c25208ecad32f7: 2023-03-29 03:53:51,281 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7., storeName=52c064cf4695189b45c25208ecad32f7/A, priority=13, startTime=1680062031191; duration=0sec 2023-03-29 03:53:51,281 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:53:51,281 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-29 03:53:51,282 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 52c064cf4695189b45c25208ecad32f7/B of 52c064cf4695189b45c25208ecad32f7 into 1f19019b749a41ed95fd3f201f6e510e(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:53:51,282 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 52c064cf4695189b45c25208ecad32f7: 2023-03-29 03:53:51,283 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7., storeName=52c064cf4695189b45c25208ecad32f7/B, priority=13, startTime=1680062031192; duration=0sec 2023-03-29 03:53:51,283 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:53:51,283 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37036 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-29 03:53:51,283 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1912): 52c064cf4695189b45c25208ecad32f7/C is initiating minor compaction (all files) 2023-03-29 03:53:51,283 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 52c064cf4695189b45c25208ecad32f7/C in TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:51,283 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/da1dc0a233214ed8bd9a00cf08938c6d, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/604081a3c79341058b1dc38ce0aa3fa6, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/8bc409548e5a42f6895639d98eb01278] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp, totalSize=36.2 K 2023-03-29 03:53:51,284 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:53:51,284 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:53:51,284 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting da1dc0a233214ed8bd9a00cf08938c6d, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=425, earliestPutTs=1680062028826 2023-03-29 03:53:51,284 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:51,284 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. started... 2023-03-29 03:53:51,284 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 52c064cf4695189b45c25208ecad32f7 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-29 03:53:51,285 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 604081a3c79341058b1dc38ce0aa3fa6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=441, earliestPutTs=1680062028944 2023-03-29 03:53:51,285 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 52c064cf4695189b45c25208ecad32f7, store=A 2023-03-29 03:53:51,285 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:53:51,285 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 52c064cf4695189b45c25208ecad32f7, store=B 2023-03-29 03:53:51,285 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:53:51,285 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 52c064cf4695189b45c25208ecad32f7, store=C 2023-03-29 03:53:51,285 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 8bc409548e5a42f6895639d98eb01278, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=461, earliestPutTs=1680062029597 2023-03-29 03:53:51,285 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:53:51,308 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] throttle.PressureAwareThroughputController(145): 52c064cf4695189b45c25208ecad32f7#C#compaction#56 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:53:51,333 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=476 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/A/e52786ef81e94ed0ba6d8fb1118466d3 2023-03-29 03:53:51,363 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=476 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/B/597d31f73e2840b59e5d2f11937c015e 2023-03-29 03:53:51,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on 52c064cf4695189b45c25208ecad32f7 2023-03-29 03:53:51,384 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. as already flushing 2023-03-29 03:53:51,411 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:51,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48240 deadline: 1680062091404, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:51,412 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:51,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48198 deadline: 1680062091408, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:51,415 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:51,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48262 deadline: 1680062091409, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:51,416 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:51,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48252 deadline: 1680062091412, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:51,417 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:51,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48182 deadline: 1680062091414, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:51,515 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:51,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48198 deadline: 1680062091514, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:51,516 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:51,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48240 deadline: 1680062091515, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:51,520 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:51,520 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:51,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48252 deadline: 1680062091518, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:51,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48262 deadline: 1680062091517, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:51,521 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:51,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48182 deadline: 1680062091518, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:51,721 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:51,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48240 deadline: 1680062091718, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:51,721 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:51,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48198 deadline: 1680062091718, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:51,724 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:51,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48262 deadline: 1680062091722, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:51,728 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:51,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48182 deadline: 1680062091723, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:51,728 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:51,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48252 deadline: 1680062091723, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:51,748 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/C/8c4cff2d682d45d984a884cde2fe1812 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/8c4cff2d682d45d984a884cde2fe1812 2023-03-29 03:53:51,764 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 52c064cf4695189b45c25208ecad32f7/C of 52c064cf4695189b45c25208ecad32f7 into 8c4cff2d682d45d984a884cde2fe1812(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:53:51,764 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 52c064cf4695189b45c25208ecad32f7: 2023-03-29 03:53:51,764 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7., storeName=52c064cf4695189b45c25208ecad32f7/C, priority=13, startTime=1680062031192; duration=0sec 2023-03-29 03:53:51,764 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:53:51,796 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=476 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/C/acff93129303436292586abe1900b05f 2023-03-29 03:53:51,803 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/A/e52786ef81e94ed0ba6d8fb1118466d3 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/e52786ef81e94ed0ba6d8fb1118466d3 2023-03-29 03:53:51,811 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/e52786ef81e94ed0ba6d8fb1118466d3, entries=150, sequenceid=476, filesize=11.9 K 2023-03-29 03:53:51,813 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/B/597d31f73e2840b59e5d2f11937c015e as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/597d31f73e2840b59e5d2f11937c015e 2023-03-29 03:53:51,833 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/597d31f73e2840b59e5d2f11937c015e, entries=150, sequenceid=476, filesize=11.9 K 2023-03-29 03:53:51,834 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/C/acff93129303436292586abe1900b05f as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/acff93129303436292586abe1900b05f 2023-03-29 03:53:51,845 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/acff93129303436292586abe1900b05f, entries=150, sequenceid=476, filesize=11.9 K 2023-03-29 03:53:51,848 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=114.05 KB/116790 for 52c064cf4695189b45c25208ecad32f7 in 564ms, sequenceid=476, compaction requested=false 2023-03-29 03:53:51,848 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 52c064cf4695189b45c25208ecad32f7: 2023-03-29 03:53:51,848 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:51,848 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:53:51,848 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:53:51,848 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:51,848 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:53:51,848 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:53:51,851 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:51,851 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:51,851 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:53:51,852 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:51,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:51,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:51,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:51,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:51,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:51,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:51,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:51,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:51,854 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:53:51,854 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@378b2f66[Count = 0] remaining members to acquire global barrier 2023-03-29 03:53:51,854 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:53:51,854 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:51,855 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:51,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:51,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:51,856 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:53:51,856 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:53:51,856 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:53:51,856 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:51,856 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:53:51,858 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:51,858 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:53:51,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:51,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:51,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:51,858 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:53:51,859 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:53:51,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:51,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:51,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:51,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:51,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:51,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:51,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:51,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:53:51,865 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:53:51,865 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:53:51,865 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:53:51,865 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:53:51,865 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:53:51,865 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:53:51,867 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:51,867 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:53:51,867 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:51,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:51,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:51,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:51,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:51,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:51,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:53:51,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:53:51,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:51,868 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:51,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:51,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:51,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:51,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:51,870 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:51,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:51,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:51,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:51,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:51,876 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:53:51,876 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:53:51,876 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:51,877 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:51,877 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:51,877 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:53:51,877 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:53:51,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:53:51,877 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:53:51,877 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:53:51,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:53:51,877 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:51,877 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:51,877 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:51,877 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:53:51,877 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:53:51,877 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:51,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:53:51,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:53:51,977 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:53:51,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:53:51,982 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:53:51,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:53:51,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:53:51,996 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:53:51,996 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:53:51,996 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:53:51,996 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:53:51,997 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:51,997 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:51,998 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:51,998 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:53:51,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:53:51,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:53:52,000 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:52,001 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:53:52,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:52,002 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:52,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:53:52,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:52,002 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:53:52,012 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:53:52,012 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:53:52,012 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:53:52,012 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:53:52,012 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:53:52,013 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:53:52,013 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:53:52,014 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:52,014 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. started... 2023-03-29 03:53:52,014 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 52c064cf4695189b45c25208ecad32f7 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-29 03:53:52,014 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 52c064cf4695189b45c25208ecad32f7, store=A 2023-03-29 03:53:52,014 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:53:52,014 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 52c064cf4695189b45c25208ecad32f7, store=B 2023-03-29 03:53:52,014 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:53:52,014 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 52c064cf4695189b45c25208ecad32f7, store=C 2023-03-29 03:53:52,014 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:53:52,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on 52c064cf4695189b45c25208ecad32f7 2023-03-29 03:53:52,038 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. as already flushing 2023-03-29 03:53:52,053 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:52,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48262 deadline: 1680062092049, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:52,054 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:52,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48240 deadline: 1680062092047, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:52,054 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:52,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48182 deadline: 1680062092051, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:52,056 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:52,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48252 deadline: 1680062092053, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:52,057 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:52,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48198 deadline: 1680062092054, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:52,158 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:52,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48182 deadline: 1680062092157, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:52,162 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:52,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48262 deadline: 1680062092157, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:52,162 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:52,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48240 deadline: 1680062092158, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:52,163 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:52,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48252 deadline: 1680062092161, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:52,164 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:52,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48198 deadline: 1680062092161, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:52,363 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:52,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48182 deadline: 1680062092361, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:52,367 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:52,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48262 deadline: 1680062092364, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:52,368 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:52,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48240 deadline: 1680062092365, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:52,368 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:52,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48252 deadline: 1680062092365, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:52,369 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:52,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48198 deadline: 1680062092366, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:52,495 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=497 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/A/65a875f9820e4483ba2a34c0d09006fc 2023-03-29 03:53:52,512 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=497 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/B/a3f166b8325a45a89018b251f74e6e78 2023-03-29 03:53:52,669 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:52,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48182 deadline: 1680062092669, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:52,675 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:52,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48198 deadline: 1680062092672, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:52,675 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:52,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48240 deadline: 1680062092672, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:52,675 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:52,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48252 deadline: 1680062092672, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:52,676 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:52,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48262 deadline: 1680062092673, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:52,926 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=497 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/C/dffced6fd44b4df3acd08c96c4275cc0 2023-03-29 03:53:52,939 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/A/65a875f9820e4483ba2a34c0d09006fc as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/65a875f9820e4483ba2a34c0d09006fc 2023-03-29 03:53:52,954 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/65a875f9820e4483ba2a34c0d09006fc, entries=150, sequenceid=497, filesize=11.9 K 2023-03-29 03:53:52,956 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/B/a3f166b8325a45a89018b251f74e6e78 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/a3f166b8325a45a89018b251f74e6e78 2023-03-29 03:53:52,967 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/a3f166b8325a45a89018b251f74e6e78, entries=150, sequenceid=497, filesize=11.9 K 2023-03-29 03:53:52,968 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/C/dffced6fd44b4df3acd08c96c4275cc0 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/dffced6fd44b4df3acd08c96c4275cc0 2023-03-29 03:53:52,997 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/dffced6fd44b4df3acd08c96c4275cc0, entries=150, sequenceid=497, filesize=11.9 K 2023-03-29 03:53:52,999 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=67.09 KB/68700 for 52c064cf4695189b45c25208ecad32f7 in 985ms, sequenceid=497, compaction requested=true 2023-03-29 03:53:52,999 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 52c064cf4695189b45c25208ecad32f7: 2023-03-29 03:53:52,999 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:52,999 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:53:52,999 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:53:52,999 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:52,999 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:53:52,999 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:53:53,001 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:53,001 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:53,002 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:53,002 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:53:53,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:53,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:53,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:53,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:53,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:53,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:53,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:53,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:53,005 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:53:53,005 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@35579808[Count = 0] remaining members to acquire global barrier 2023-03-29 03:53:53,005 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:53:53,005 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:53,007 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:53,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:53,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:53,007 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:53:53,007 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:53:53,007 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:53:53,008 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:53,008 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:53:53,009 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:53,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:53,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:53,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:53,009 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:53:53,010 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:53:53,010 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:53:53,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:53,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:53,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:53,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:53,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:53,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:53,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:53,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:53:53,014 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:53:53,014 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:53:53,014 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:53:53,014 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:53:53,015 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:53:53,015 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:53:53,021 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:53,021 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:53,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:53,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:53,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:53,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:53,021 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:53:53,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:53,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:53,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:53:53,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:53:53,022 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:53,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:53,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:53,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:53,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:53,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:53,025 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:53,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:53,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:53,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:53,033 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:53,033 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:53,033 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:53,033 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:53,033 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:53:53,033 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:53:53,033 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:53,033 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:53:53,033 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:53:53,033 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:53:53,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:53:53,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:53:53,033 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:53:53,034 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:53:53,034 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:53:53,033 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:53,034 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:53,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:53:53,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:53:53,134 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:53:53,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:53:53,139 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:53:53,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:53:53,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:53:53,152 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:53:53,152 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:53:53,152 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:53:53,152 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:53:53,153 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:53,153 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:53,154 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:53,154 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:53:53,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:53:53,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:53:53,155 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:53,155 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:53:53,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:53,156 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:53,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:53:53,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:53,156 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:53:53,164 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:53:53,164 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:53:53,164 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:53:53,164 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:53:53,164 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:53:53,164 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:53:53,164 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:53:53,164 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:53,165 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. started... 2023-03-29 03:53:53,165 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 52c064cf4695189b45c25208ecad32f7 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-29 03:53:53,165 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 52c064cf4695189b45c25208ecad32f7, store=A 2023-03-29 03:53:53,165 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:53:53,165 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 52c064cf4695189b45c25208ecad32f7, store=B 2023-03-29 03:53:53,165 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:53:53,165 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 52c064cf4695189b45c25208ecad32f7, store=C 2023-03-29 03:53:53,166 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:53:53,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on 52c064cf4695189b45c25208ecad32f7 2023-03-29 03:53:53,178 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. as already flushing 2023-03-29 03:53:53,218 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=510 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/A/c7fbb355302e422c90ada27b07f1d0aa 2023-03-29 03:53:53,219 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:53,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48262 deadline: 1680062093208, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:53,219 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:53,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48240 deadline: 1680062093209, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:53,220 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:53,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48182 deadline: 1680062093212, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:53,224 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:53,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48252 deadline: 1680062093219, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:53,224 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:53,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48198 deadline: 1680062093219, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:53,285 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=510 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/B/c71cf56c2a3f4e66929198d3dd637832 2023-03-29 03:53:53,332 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:53,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48252 deadline: 1680062093327, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:53,341 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:53,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48198 deadline: 1680062093327, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:53,342 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:53,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48182 deadline: 1680062093327, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:53,343 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:53,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48262 deadline: 1680062093327, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:53,343 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:53,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48240 deadline: 1680062093327, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:53,404 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=510 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/C/edc9218aaf8c4c96880b7c1380a6d72f 2023-03-29 03:53:53,414 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/A/c7fbb355302e422c90ada27b07f1d0aa as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/c7fbb355302e422c90ada27b07f1d0aa 2023-03-29 03:53:53,431 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/c7fbb355302e422c90ada27b07f1d0aa, entries=150, sequenceid=510, filesize=11.9 K 2023-03-29 03:53:53,433 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/B/c71cf56c2a3f4e66929198d3dd637832 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/c71cf56c2a3f4e66929198d3dd637832 2023-03-29 03:53:53,442 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/c71cf56c2a3f4e66929198d3dd637832, entries=150, sequenceid=510, filesize=11.9 K 2023-03-29 03:53:53,451 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/C/edc9218aaf8c4c96880b7c1380a6d72f as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/edc9218aaf8c4c96880b7c1380a6d72f 2023-03-29 03:53:53,461 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/edc9218aaf8c4c96880b7c1380a6d72f, entries=150, sequenceid=510, filesize=11.9 K 2023-03-29 03:53:53,463 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 52c064cf4695189b45c25208ecad32f7 in 298ms, sequenceid=510, compaction requested=true 2023-03-29 03:53:53,463 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 52c064cf4695189b45c25208ecad32f7: 2023-03-29 03:53:53,463 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:53,463 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:53:53,463 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:53:53,463 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:53,464 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:53:53,464 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:53:53,467 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:53,467 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:53,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:53,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:53,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:53,467 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:53,467 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:53:53,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:53,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:53,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:53,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:53,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:53,469 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:53:53,469 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@429974fc[Count = 0] remaining members to acquire global barrier 2023-03-29 03:53:53,469 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:53:53,469 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:53,470 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:53,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:53,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:53,471 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:53:53,471 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:53:53,471 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:53:53,471 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:53,471 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:53:53,474 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:53,474 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:53:53,474 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:53:53,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:53,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:53,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:53,474 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:53:53,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:53,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:53,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:53,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:53,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:53,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:53,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:53,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:53:53,477 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:53:53,477 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:53:53,477 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:53:53,477 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:53:53,477 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:53:53,477 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:53:53,479 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:53,479 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:53,479 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:53:53,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:53,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:53,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:53,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:53,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:53,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:53,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:53:53,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:53:53,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:53,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:53,481 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:53,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:53,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:53,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:53,483 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:53,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:53,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:53,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:53,488 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:53,488 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:53:53,488 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:53:53,488 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:53:53,488 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:53:53,488 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:53:53,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:53:53,488 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:53,488 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:53,488 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:53,488 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:53,488 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:53,488 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:53,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:53:53,488 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:53:53,489 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:53:53,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:53:53,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:53:53,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on 52c064cf4695189b45c25208ecad32f7 2023-03-29 03:53:53,543 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 52c064cf4695189b45c25208ecad32f7 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-29 03:53:53,544 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 52c064cf4695189b45c25208ecad32f7, store=A 2023-03-29 03:53:53,544 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:53:53,544 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 52c064cf4695189b45c25208ecad32f7, store=B 2023-03-29 03:53:53,544 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:53:53,544 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 52c064cf4695189b45c25208ecad32f7, store=C 2023-03-29 03:53:53,544 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:53:53,560 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:53,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48262 deadline: 1680062093556, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:53,570 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=530 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/A/1914264028cf4cf786c390e2ff8fb7fd 2023-03-29 03:53:53,570 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:53,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48182 deadline: 1680062093558, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:53,571 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:53,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48198 deadline: 1680062093559, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:53,571 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:53,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48240 deadline: 1680062093560, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:53,570 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:53,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48252 deadline: 1680062093558, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:53,589 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:53:53,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:53:53,594 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:53:53,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:53:53,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:53:53,604 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=530 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/B/eb6aa41f262e45aa833562685b63e7ed 2023-03-29 03:53:53,604 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:53:53,604 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:53:53,604 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:53:53,604 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:53:53,605 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:53,605 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:53,606 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:53:53,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:53:53,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:53:53,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:53,607 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:53,608 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:53,608 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:53,608 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:53:53,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:53:53,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:53,608 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:53:53,609 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:53:53,609 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:53:53,609 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:53:53,609 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:53:53,609 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:53:53,609 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:53:53,609 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:53:53,609 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:53,609 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. started... 2023-03-29 03:53:53,609 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. as already flushing 2023-03-29 03:53:53,609 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:53,609 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:53:53,609 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:53:53,610 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:53,610 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:53:53,610 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:53:53,611 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:53,611 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:53,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:53,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:53,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:53,612 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:53,612 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:53:53,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:53,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:53,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:53,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:53,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:53,615 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:53:53,615 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f4559a4[Count = 0] remaining members to acquire global barrier 2023-03-29 03:53:53,615 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:53:53,615 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:53,617 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:53,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:53,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:53,617 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:53:53,617 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:53:53,617 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:53:53,617 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:53,617 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:53:53,619 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:53,619 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:53:53,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:53,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:53,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:53,619 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:53:53,620 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:53:53,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:53,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:53,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:53,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:53,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:53,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:53,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:53,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:53:53,624 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:53:53,624 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:53:53,624 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:53:53,624 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:53:53,624 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:53:53,625 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:53:53,627 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:53,627 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:53:53,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:53,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:53,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:53:53,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:53:53,628 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:53,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:53,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:53,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:53,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:53,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:53,629 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:53,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:53,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:53,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:53,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:53,631 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:53,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:53,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:53,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:53,634 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:53:53,634 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:53:53,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:53:53,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:53:53,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:53:53,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:53:53,636 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:53,636 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:53,636 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:53,636 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:53:53,636 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:53,636 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:53,636 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:53:53,636 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:53:53,636 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:53,636 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:53,636 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:53:53,636 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:53:53,639 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=530 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/C/b58c2f5b9a924dac9a935c16f16870bf 2023-03-29 03:53:53,645 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/A/1914264028cf4cf786c390e2ff8fb7fd as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/1914264028cf4cf786c390e2ff8fb7fd 2023-03-29 03:53:53,661 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/1914264028cf4cf786c390e2ff8fb7fd, entries=200, sequenceid=530, filesize=14.3 K 2023-03-29 03:53:53,662 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/B/eb6aa41f262e45aa833562685b63e7ed as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/eb6aa41f262e45aa833562685b63e7ed 2023-03-29 03:53:53,667 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:53,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48262 deadline: 1680062093662, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:53,669 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/eb6aa41f262e45aa833562685b63e7ed, entries=150, sequenceid=530, filesize=11.9 K 2023-03-29 03:53:53,670 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/C/b58c2f5b9a924dac9a935c16f16870bf as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/b58c2f5b9a924dac9a935c16f16870bf 2023-03-29 03:53:53,673 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:53,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48182 deadline: 1680062093672, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:53,673 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:53,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48198 deadline: 1680062093673, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:53,674 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:53,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48240 deadline: 1680062093673, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:53,674 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:53,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48252 deadline: 1680062093673, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:53,678 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/b58c2f5b9a924dac9a935c16f16870bf, entries=150, sequenceid=530, filesize=11.9 K 2023-03-29 03:53:53,679 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 52c064cf4695189b45c25208ecad32f7 in 136ms, sequenceid=530, compaction requested=true 2023-03-29 03:53:53,679 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 52c064cf4695189b45c25208ecad32f7: 2023-03-29 03:53:53,679 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:53:53,679 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-29 03:53:53,679 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-29 03:53:53,679 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:53:53,679 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:53:53,682 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61544 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-29 03:53:53,682 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 63984 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-29 03:53:53,682 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1912): 52c064cf4695189b45c25208ecad32f7/B is initiating minor compaction (all files) 2023-03-29 03:53:53,682 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1912): 52c064cf4695189b45c25208ecad32f7/A is initiating minor compaction (all files) 2023-03-29 03:53:53,682 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 52c064cf4695189b45c25208ecad32f7/B in TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:53,682 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 52c064cf4695189b45c25208ecad32f7/A in TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:53,682 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/1f19019b749a41ed95fd3f201f6e510e, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/597d31f73e2840b59e5d2f11937c015e, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/a3f166b8325a45a89018b251f74e6e78, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/c71cf56c2a3f4e66929198d3dd637832, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/eb6aa41f262e45aa833562685b63e7ed] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp, totalSize=60.1 K 2023-03-29 03:53:53,682 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/a61dc4c8d4f2402db89d0911daeef838, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/e52786ef81e94ed0ba6d8fb1118466d3, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/65a875f9820e4483ba2a34c0d09006fc, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/c7fbb355302e422c90ada27b07f1d0aa, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/1914264028cf4cf786c390e2ff8fb7fd] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp, totalSize=62.5 K 2023-03-29 03:53:53,683 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 1f19019b749a41ed95fd3f201f6e510e, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=461, earliestPutTs=1680062029597 2023-03-29 03:53:53,683 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting a61dc4c8d4f2402db89d0911daeef838, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=461, earliestPutTs=1680062029597 2023-03-29 03:53:53,683 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 597d31f73e2840b59e5d2f11937c015e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=476, earliestPutTs=1680062030216 2023-03-29 03:53:53,683 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting e52786ef81e94ed0ba6d8fb1118466d3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=476, earliestPutTs=1680062030216 2023-03-29 03:53:53,684 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting a3f166b8325a45a89018b251f74e6e78, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=497, earliestPutTs=1680062031402 2023-03-29 03:53:53,684 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 65a875f9820e4483ba2a34c0d09006fc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=497, earliestPutTs=1680062031402 2023-03-29 03:53:53,685 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting c71cf56c2a3f4e66929198d3dd637832, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=510, earliestPutTs=1680062032045 2023-03-29 03:53:53,686 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting c7fbb355302e422c90ada27b07f1d0aa, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=510, earliestPutTs=1680062032045 2023-03-29 03:53:53,686 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting eb6aa41f262e45aa833562685b63e7ed, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=530, earliestPutTs=1680062033206 2023-03-29 03:53:53,686 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 1914264028cf4cf786c390e2ff8fb7fd, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=530, earliestPutTs=1680062033205 2023-03-29 03:53:53,710 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] throttle.PressureAwareThroughputController(145): 52c064cf4695189b45c25208ecad32f7#A#compaction#68 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:53:53,714 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] throttle.PressureAwareThroughputController(145): 52c064cf4695189b45c25208ecad32f7#B#compaction#69 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:53:53,736 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:53:53,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:53:53,740 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:53:53,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:53:53,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:53:53,767 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:53:53,767 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:53:53,767 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:53:53,767 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:53:53,767 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:53,767 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:53,769 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:53,769 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:53:53,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:53:53,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:53:53,769 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:53,769 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:53:53,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:53,770 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:53,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:53:53,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:53,770 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:53:53,784 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:53:53,784 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:53:53,784 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:53:53,784 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:53:53,784 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:53:53,788 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:53:53,788 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:53:53,788 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:53,788 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. started... 2023-03-29 03:53:53,796 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 52c064cf4695189b45c25208ecad32f7 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-29 03:53:53,796 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 52c064cf4695189b45c25208ecad32f7, store=A 2023-03-29 03:53:53,796 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:53:53,796 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 52c064cf4695189b45c25208ecad32f7, store=B 2023-03-29 03:53:53,796 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:53:53,797 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 52c064cf4695189b45c25208ecad32f7, store=C 2023-03-29 03:53:53,797 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:53:53,800 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/A/90da68bdf8444aaba4f996f746916e58 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/90da68bdf8444aaba4f996f746916e58 2023-03-29 03:53:53,808 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 52c064cf4695189b45c25208ecad32f7/A of 52c064cf4695189b45c25208ecad32f7 into 90da68bdf8444aaba4f996f746916e58(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:53:53,808 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 52c064cf4695189b45c25208ecad32f7: 2023-03-29 03:53:53,808 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7., storeName=52c064cf4695189b45c25208ecad32f7/A, priority=11, startTime=1680062033679; duration=0sec 2023-03-29 03:53:53,808 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:53:53,808 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-29 03:53:53,810 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61510 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-29 03:53:53,810 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1912): 52c064cf4695189b45c25208ecad32f7/C is initiating minor compaction (all files) 2023-03-29 03:53:53,810 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 52c064cf4695189b45c25208ecad32f7/C in TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:53,810 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/8c4cff2d682d45d984a884cde2fe1812, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/acff93129303436292586abe1900b05f, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/dffced6fd44b4df3acd08c96c4275cc0, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/edc9218aaf8c4c96880b7c1380a6d72f, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/b58c2f5b9a924dac9a935c16f16870bf] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp, totalSize=60.1 K 2023-03-29 03:53:53,811 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 8c4cff2d682d45d984a884cde2fe1812, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=461, earliestPutTs=1680062029597 2023-03-29 03:53:53,811 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting acff93129303436292586abe1900b05f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=476, earliestPutTs=1680062030216 2023-03-29 03:53:53,811 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting dffced6fd44b4df3acd08c96c4275cc0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=497, earliestPutTs=1680062031402 2023-03-29 03:53:53,812 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting edc9218aaf8c4c96880b7c1380a6d72f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=510, earliestPutTs=1680062032045 2023-03-29 03:53:53,812 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting b58c2f5b9a924dac9a935c16f16870bf, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=530, earliestPutTs=1680062033206 2023-03-29 03:53:53,826 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] throttle.PressureAwareThroughputController(145): 52c064cf4695189b45c25208ecad32f7#C#compaction#71 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:53:53,848 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/C/8a8dbebf193043dcaa8bb69122bbc562 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/8a8dbebf193043dcaa8bb69122bbc562 2023-03-29 03:53:53,854 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 52c064cf4695189b45c25208ecad32f7/C of 52c064cf4695189b45c25208ecad32f7 into 8a8dbebf193043dcaa8bb69122bbc562(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:53:53,855 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 52c064cf4695189b45c25208ecad32f7: 2023-03-29 03:53:53,855 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7., storeName=52c064cf4695189b45c25208ecad32f7/C, priority=11, startTime=1680062033679; duration=0sec 2023-03-29 03:53:53,855 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:53:53,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on 52c064cf4695189b45c25208ecad32f7 2023-03-29 03:53:53,870 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. as already flushing 2023-03-29 03:53:53,897 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:53,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48182 deadline: 1680062093891, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:53,897 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:53,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48198 deadline: 1680062093892, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:53,898 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1680062033890, max timestamp=1680062033895Num uniques 859; 2023-03-29 03:53:53,898 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-29 03:53:53,898 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1680062033890, max timestamp=1680062033895Num uniques 859; 2023-03-29 03:53:53,898 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-29 03:53:53,898 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1680062033890, max timestamp=1680062033895Num uniques 859; 2023-03-29 03:53:53,899 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-29 03:53:53,958 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:53,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48252 deadline: 1680062093952, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:53,959 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:53,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48240 deadline: 1680062093952, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:53,959 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:53,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48262 deadline: 1680062093952, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:54,001 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:54,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48182 deadline: 1680062093999, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:54,002 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:54,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48198 deadline: 1680062093999, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:54,061 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:54,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48252 deadline: 1680062094060, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:54,064 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:54,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48240 deadline: 1680062094060, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:54,064 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:54,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48262 deadline: 1680062094060, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:54,203 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/B/698b56d2bd50460ba87ac4df94c10b2f as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/698b56d2bd50460ba87ac4df94c10b2f 2023-03-29 03:53:54,209 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:54,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48198 deadline: 1680062094203, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:54,209 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:54,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48182 deadline: 1680062094204, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:54,212 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 52c064cf4695189b45c25208ecad32f7/B of 52c064cf4695189b45c25208ecad32f7 into 698b56d2bd50460ba87ac4df94c10b2f(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:53:54,213 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 52c064cf4695189b45c25208ecad32f7: 2023-03-29 03:53:54,213 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7., storeName=52c064cf4695189b45c25208ecad32f7/B, priority=11, startTime=1680062033679; duration=0sec 2023-03-29 03:53:54,213 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:53:54,215 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=542 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/A/396832ffc8ab44adb5c19073682d4869 2023-03-29 03:53:54,265 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:54,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48252 deadline: 1680062094263, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:54,268 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:54,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48240 deadline: 1680062094266, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:54,269 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:54,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48262 deadline: 1680062094268, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:54,281 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=542 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/B/ec472197725345a3ab0b77e6424362ef 2023-03-29 03:53:54,513 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:54,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48198 deadline: 1680062094512, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:54,514 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:54,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48182 deadline: 1680062094512, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:54,570 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:54,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48252 deadline: 1680062094569, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:54,575 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:54,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48240 deadline: 1680062094573, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:54,575 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:54,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48262 deadline: 1680062094574, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:54,715 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=542 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/C/00bd1303873a4740b15d08e631aac433 2023-03-29 03:53:54,722 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/A/396832ffc8ab44adb5c19073682d4869 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/396832ffc8ab44adb5c19073682d4869 2023-03-29 03:53:54,728 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/396832ffc8ab44adb5c19073682d4869, entries=150, sequenceid=542, filesize=11.9 K 2023-03-29 03:53:54,729 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/B/ec472197725345a3ab0b77e6424362ef as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/ec472197725345a3ab0b77e6424362ef 2023-03-29 03:53:54,737 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/ec472197725345a3ab0b77e6424362ef, entries=150, sequenceid=542, filesize=11.9 K 2023-03-29 03:53:54,739 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/C/00bd1303873a4740b15d08e631aac433 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/00bd1303873a4740b15d08e631aac433 2023-03-29 03:53:54,751 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/00bd1303873a4740b15d08e631aac433, entries=150, sequenceid=542, filesize=11.9 K 2023-03-29 03:53:54,754 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=134.18 KB/137400 for 52c064cf4695189b45c25208ecad32f7 in 958ms, sequenceid=542, compaction requested=false 2023-03-29 03:53:54,754 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 52c064cf4695189b45c25208ecad32f7: 2023-03-29 03:53:54,754 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:54,754 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:53:54,754 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:53:54,754 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:54,754 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:53:54,754 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:53:54,756 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:54,756 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:54,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:54,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:54,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:54,757 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:54,757 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:53:54,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:54,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:54,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:54,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:54,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:54,760 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:53:54,760 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@21dbbf35[Count = 0] remaining members to acquire global barrier 2023-03-29 03:53:54,760 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:53:54,760 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:54,761 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:54,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:54,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:54,762 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:53:54,762 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:53:54,762 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:53:54,762 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:54,762 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:53:54,763 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:53:54,763 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:53:54,764 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:53:54,764 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:54,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:54,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:54,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:54,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:54,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:54,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:54,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:54,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:54,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:54,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:54,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:53:54,768 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:53:54,768 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:53:54,768 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:53:54,768 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:53:54,768 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:53:54,768 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:53:54,771 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:54,771 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:54,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:54,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:54,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:54,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:54,771 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:53:54,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:54,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:54,772 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:54,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:53:54,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:53:54,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:54,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:54,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:54,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:54,773 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:54,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:54,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:54,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:54,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:54,776 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:54,776 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:54,776 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:54,776 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:53:54,776 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:53:54,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:53:54,776 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:54,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:53:54,777 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:54,777 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:53:54,777 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:53:54,777 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:54,777 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:54,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:53:54,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:53:54,777 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:53:54,777 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:53:54,777 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:53:54,777 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:53:54,878 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:53:54,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:53:54,880 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:53:54,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:53:54,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:53:54,882 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:53:54,883 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:53:54,883 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:53:54,883 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:53:54,883 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:54,883 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:54,887 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:53:54,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:53:54,887 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:54,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:53:54,888 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:54,888 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:53:54,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:54,888 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:54,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:53:54,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:54,889 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:53:54,889 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:53:54,889 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:53:54,889 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:53:54,889 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:53:54,889 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:53:54,889 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:53:54,889 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:53:54,889 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:54,889 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. started... 2023-03-29 03:53:54,890 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 52c064cf4695189b45c25208ecad32f7 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-03-29 03:53:54,890 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 52c064cf4695189b45c25208ecad32f7, store=A 2023-03-29 03:53:54,890 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-29 03:53:54,890 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 52c064cf4695189b45c25208ecad32f7, store=B 2023-03-29 03:53:54,890 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-29 03:53:54,890 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 52c064cf4695189b45c25208ecad32f7, store=C 2023-03-29 03:53:54,890 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-29 03:53:55,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on 52c064cf4695189b45c25208ecad32f7 2023-03-29 03:53:55,020 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. as already flushing 2023-03-29 03:53:55,032 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:55,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48182 deadline: 1680062095029, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:55,033 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:55,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48198 deadline: 1680062095032, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:55,076 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:55,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48252 deadline: 1680062095075, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:55,079 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:55,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48262 deadline: 1680062095079, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:55,083 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:55,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48240 deadline: 1680062095080, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:55,137 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:55,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48182 deadline: 1680062095133, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:55,139 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:55,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48198 deadline: 1680062095135, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:55,300 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=581 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/A/0ecd46374c9747fcad6808aacf58beed 2023-03-29 03:53:55,333 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=581 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/B/4e23e12eee9946de95509c2a1c524e95 2023-03-29 03:53:55,340 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:55,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48182 deadline: 1680062095339, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:55,345 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:55,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48198 deadline: 1680062095342, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:55,381 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=581 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/C/8e63a611e642470f849de454861c88db 2023-03-29 03:53:55,388 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/A/0ecd46374c9747fcad6808aacf58beed as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/0ecd46374c9747fcad6808aacf58beed 2023-03-29 03:53:55,394 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/0ecd46374c9747fcad6808aacf58beed, entries=150, sequenceid=581, filesize=11.9 K 2023-03-29 03:53:55,395 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/B/4e23e12eee9946de95509c2a1c524e95 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/4e23e12eee9946de95509c2a1c524e95 2023-03-29 03:53:55,401 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/4e23e12eee9946de95509c2a1c524e95, entries=150, sequenceid=581, filesize=11.9 K 2023-03-29 03:53:55,402 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/C/8e63a611e642470f849de454861c88db as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/8e63a611e642470f849de454861c88db 2023-03-29 03:53:55,417 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/8e63a611e642470f849de454861c88db, entries=150, sequenceid=581, filesize=11.9 K 2023-03-29 03:53:55,418 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for 52c064cf4695189b45c25208ecad32f7 in 529ms, sequenceid=581, compaction requested=true 2023-03-29 03:53:55,418 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 52c064cf4695189b45c25208ecad32f7: 2023-03-29 03:53:55,418 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:55,418 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:53:55,418 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:53:55,418 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:55,418 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:53:55,418 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:53:55,420 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:55,420 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:55,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:55,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:55,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:55,421 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:55,421 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:53:55,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:55,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:55,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:55,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:55,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:55,422 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:53:55,422 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@28c5f4de[Count = 0] remaining members to acquire global barrier 2023-03-29 03:53:55,422 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:53:55,422 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:55,425 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:55,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:55,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:55,425 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:53:55,425 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:53:55,425 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:53:55,426 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:55,426 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:53:55,427 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:53:55,427 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:53:55,427 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:55,427 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:53:55,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:55,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:55,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:55,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:55,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:55,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:55,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:55,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:55,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:55,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:55,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:53:55,432 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:53:55,432 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:53:55,432 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:53:55,432 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:53:55,432 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:53:55,432 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:53:55,434 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:55,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:55,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:55,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:55,434 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:55,434 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:53:55,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:55,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:55,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:55,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:53:55,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:55,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:53:55,435 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:55,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:55,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:55,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:55,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:55,437 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:55,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:55,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:55,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:55,440 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:55,440 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:53:55,440 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:53:55,440 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:53:55,440 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:55,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:53:55,440 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:55,440 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:53:55,440 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:53:55,440 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:53:55,440 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:55,441 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:55,441 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:55,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:53:55,441 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:55,441 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:53:55,441 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:53:55,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:53:55,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:53:55,541 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:53:55,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:53:55,544 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:53:55,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:53:55,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:53:55,549 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:53:55,549 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:53:55,549 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:53:55,549 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:53:55,550 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:55,550 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:55,552 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:53:55,552 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:55,552 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:53:55,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:53:55,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:55,553 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:55,553 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:53:55,553 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:55,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:53:55,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:55,554 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:53:55,554 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:53:55,554 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:53:55,554 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:53:55,554 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:53:55,554 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:53:55,560 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:53:55,561 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:55,561 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. started... 2023-03-29 03:53:55,561 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:53:55,561 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 52c064cf4695189b45c25208ecad32f7 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-29 03:53:55,561 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 52c064cf4695189b45c25208ecad32f7, store=A 2023-03-29 03:53:55,561 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:53:55,561 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 52c064cf4695189b45c25208ecad32f7, store=B 2023-03-29 03:53:55,561 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:53:55,561 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 52c064cf4695189b45c25208ecad32f7, store=C 2023-03-29 03:53:55,561 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:53:55,571 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=591 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/A/c0a13627cc4844f2a33f9f49d8c26510 2023-03-29 03:53:55,592 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=591 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/B/7d775e277608481dae58badfd82ae313 2023-03-29 03:53:55,613 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=591 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/C/05eb172415b84dd3b84da4f0797a2507 2023-03-29 03:53:55,620 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/A/c0a13627cc4844f2a33f9f49d8c26510 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/c0a13627cc4844f2a33f9f49d8c26510 2023-03-29 03:53:55,625 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/c0a13627cc4844f2a33f9f49d8c26510, entries=150, sequenceid=591, filesize=11.9 K 2023-03-29 03:53:55,626 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/B/7d775e277608481dae58badfd82ae313 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/7d775e277608481dae58badfd82ae313 2023-03-29 03:53:55,640 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/7d775e277608481dae58badfd82ae313, entries=150, sequenceid=591, filesize=11.9 K 2023-03-29 03:53:55,641 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/C/05eb172415b84dd3b84da4f0797a2507 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/05eb172415b84dd3b84da4f0797a2507 2023-03-29 03:53:55,648 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/05eb172415b84dd3b84da4f0797a2507, entries=150, sequenceid=591, filesize=11.9 K 2023-03-29 03:53:55,650 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=6.71 KB/6870 for 52c064cf4695189b45c25208ecad32f7 in 89ms, sequenceid=591, compaction requested=true 2023-03-29 03:53:55,650 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 52c064cf4695189b45c25208ecad32f7: 2023-03-29 03:53:55,650 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:55,650 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:53:55,650 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:53:55,650 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:55,654 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:53:55,654 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:53:55,656 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:55,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:55,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:55,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:55,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:55,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:55,657 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:55,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:55,657 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:55,658 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:53:55,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:55,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:55,659 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:53:55,660 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@21724bae[Count = 0] remaining members to acquire global barrier 2023-03-29 03:53:55,660 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:53:55,660 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:55,661 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:55,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:55,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:55,661 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:53:55,661 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:53:55,661 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:53:55,662 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:55,662 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:53:55,663 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:53:55,663 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:53:55,663 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:53:55,663 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:55,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:55,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:55,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:55,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:55,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:55,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:55,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:55,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:55,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:55,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:55,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:53:55,672 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:53:55,672 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:53:55,672 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:53:55,672 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:53:55,672 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:53:55,672 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:53:55,675 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:55,675 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:53:55,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:55,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:55,676 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:55,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:55,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:55,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:55,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:53:55,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:53:55,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:55,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:55,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:55,678 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:55,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:55,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:55,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:55,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:55,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:55,681 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:55,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:55,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on 52c064cf4695189b45c25208ecad32f7 2023-03-29 03:53:55,689 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 52c064cf4695189b45c25208ecad32f7 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-29 03:53:55,689 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 52c064cf4695189b45c25208ecad32f7, store=A 2023-03-29 03:53:55,689 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:53:55,689 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 52c064cf4695189b45c25208ecad32f7, store=B 2023-03-29 03:53:55,689 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:53:55,689 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 52c064cf4695189b45c25208ecad32f7, store=C 2023-03-29 03:53:55,689 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:53:55,691 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:55,691 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:53:55,691 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:53:55,691 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:53:55,691 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:53:55,691 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:55,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:53:55,691 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:53:55,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:53:55,691 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:53:55,691 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:53:55,691 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:55,691 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:55,691 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:55,691 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:55,691 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:55,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:53:55,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:53:55,745 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=601 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/A/12d7b13eb7054aef8d0a5eb7197bcc94 2023-03-29 03:53:55,781 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680062035738, max timestamp=1680062035779Num uniques 859; 2023-03-29 03:53:55,781 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680062035738, max timestamp=1680062035779Num uniques 859; 2023-03-29 03:53:55,781 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-29 03:53:55,781 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-29 03:53:55,783 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680062035738, max timestamp=1680062035779Num uniques 859; 2023-03-29 03:53:55,783 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-29 03:53:55,791 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:53:55,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:53:55,795 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:53:55,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:53:55,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:53:55,840 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:53:55,840 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:53:55,840 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:53:55,840 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:53:55,841 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:55,841 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:55,843 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:53:55,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:53:55,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:53:55,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:55,844 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:55,845 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:55,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:53:55,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:55,845 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:53:55,846 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:55,846 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:53:55,849 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:53:55,849 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:53:55,849 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:53:55,849 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:53:55,849 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:53:55,850 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:53:55,850 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:53:55,850 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:55,850 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. started... 2023-03-29 03:53:55,850 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. as already flushing 2023-03-29 03:53:55,850 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:55,850 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:53:55,850 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:53:55,850 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:55,851 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:53:55,851 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:53:55,851 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:55,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48198 deadline: 1680062095846, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:55,852 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:55,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48182 deadline: 1680062095847, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:55,855 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:55,855 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:55,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:55,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:55,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:55,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:55,855 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:55,855 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:53:55,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:55,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:55,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:55,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:55,858 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:53:55,858 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@449fbe5d[Count = 0] remaining members to acquire global barrier 2023-03-29 03:53:55,858 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:53:55,858 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:55,859 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:55,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:55,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:55,860 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:53:55,860 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:53:55,860 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:53:55,860 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:55,860 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:53:55,861 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:55,861 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:53:55,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:55,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:55,861 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:53:55,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:55,861 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:53:55,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:55,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:55,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:55,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:55,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:55,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:55,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:55,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:53:55,864 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:53:55,864 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:53:55,864 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:53:55,864 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:53:55,864 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:53:55,864 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:53:55,867 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:55,867 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:53:55,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:55,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:55,867 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:55,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:55,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:55,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:55,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:53:55,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:53:55,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:55,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:55,868 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:55,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:55,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:55,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:55,870 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:55,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:55,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:55,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:55,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:55,875 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:55,875 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:55,875 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:55,875 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:55,875 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:55,875 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:53:55,875 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:53:55,875 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:53:55,875 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:53:55,875 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:53:55,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:53:55,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:53:55,875 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:53:55,875 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:53:55,875 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:55,875 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:55,875 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:53:55,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:53:55,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:53:55,958 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:55,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48182 deadline: 1680062095953, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:55,958 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:55,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48198 deadline: 1680062095953, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:55,975 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:53:55,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:53:55,978 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:53:55,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:53:55,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:53:55,979 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:53:55,979 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:53:55,979 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:53:55,979 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:53:55,980 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:55,980 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:55,981 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:53:55,981 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:55,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:53:55,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:53:55,982 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:55,982 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:53:55,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:55,982 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:55,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:53:55,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:55,982 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:53:55,983 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:53:55,983 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:53:55,983 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:53:55,983 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:53:55,983 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:53:55,984 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:53:55,984 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:53:55,984 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:55,984 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. started... 2023-03-29 03:53:55,984 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. as already flushing 2023-03-29 03:53:55,984 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:55,984 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:53:55,984 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:53:55,984 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:55,984 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:53:55,984 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:53:55,986 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:55,986 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:55,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:55,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:55,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:55,986 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:55,986 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:53:55,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:55,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:55,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:55,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:55,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:55,988 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:53:55,988 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@384183ec[Count = 0] remaining members to acquire global barrier 2023-03-29 03:53:55,988 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:53:55,988 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:55,992 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:55,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:55,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:55,992 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:53:55,992 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:53:55,992 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:53:55,992 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:55,992 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:53:55,994 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:55,994 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:53:55,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:55,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:55,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:55,994 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:53:55,994 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:53:55,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:55,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:55,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:55,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:55,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:55,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:55,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:55,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:53:55,996 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:53:55,996 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:53:55,996 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:53:55,996 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:53:55,996 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:53:55,996 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:53:55,998 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:55,998 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:55,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:55,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:55,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:55,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:55,998 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:53:55,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:55,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:53:55,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:53:55,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:55,999 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:55,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:55,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:55,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:56,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:56,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:56,000 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:56,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:56,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:56,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:56,002 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:56,002 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:53:56,002 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:56,002 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:53:56,002 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:53:56,002 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:56,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:53:56,002 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:53:56,002 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:53:56,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:53:56,002 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:56,003 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:53:56,003 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:56,003 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:53:56,003 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:56,003 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:56,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:53:56,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:53:56,086 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:56,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48262 deadline: 1680062096084, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:56,087 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:56,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48252 deadline: 1680062096086, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:56,092 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:56,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48240 deadline: 1680062096089, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:56,103 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:53:56,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:53:56,105 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:53:56,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:53:56,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:53:56,106 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:53:56,107 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:53:56,107 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:53:56,107 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:53:56,107 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:56,107 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:56,109 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:53:56,109 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:56,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:53:56,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:53:56,109 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:56,110 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:53:56,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:56,110 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:56,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:53:56,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:56,110 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:53:56,110 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:53:56,110 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:53:56,110 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:53:56,111 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:53:56,111 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:53:56,111 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:53:56,111 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:53:56,111 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:56,111 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. started... 2023-03-29 03:53:56,111 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. as already flushing 2023-03-29 03:53:56,111 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:56,111 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:53:56,111 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:53:56,111 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:56,111 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:53:56,111 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:53:56,113 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:56,113 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:56,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:56,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:56,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:56,113 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:56,113 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:53:56,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:56,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:56,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:56,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:56,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:56,115 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:53:56,115 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3fa5f33e[Count = 0] remaining members to acquire global barrier 2023-03-29 03:53:56,115 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:53:56,115 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:56,116 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:56,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:56,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:56,116 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:53:56,116 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:53:56,116 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:56,116 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:53:56,116 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:53:56,118 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:56,118 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:53:56,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:56,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:56,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:56,118 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:53:56,118 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:53:56,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:56,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:56,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:56,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:56,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:56,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:56,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:56,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:53:56,121 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:53:56,121 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:53:56,121 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:53:56,121 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:53:56,121 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:53:56,121 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:53:56,124 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:56,124 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:56,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:56,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:56,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:56,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:56,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:56,124 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:53:56,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:56,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:53:56,125 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:56,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:53:56,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:56,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:56,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:56,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:56,126 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:56,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:56,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:56,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:56,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:56,128 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:56,129 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:56,129 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:56,129 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:56,129 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:53:56,129 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:53:56,129 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:53:56,129 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:53:56,129 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:56,129 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:53:56,129 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:53:56,129 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:56,129 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:56,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:53:56,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:53:56,129 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:53:56,129 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:53:56,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:53:56,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:53:56,161 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:56,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48182 deadline: 1680062096160, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:56,162 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:56,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48198 deadline: 1680062096161, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:56,224 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=601 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/B/530a965f47df4a2fa1336e533d825a7b 2023-03-29 03:53:56,229 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:53:56,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:53:56,232 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:53:56,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:53:56,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:53:56,233 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:53:56,233 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:53:56,233 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:53:56,234 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:53:56,234 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:56,234 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:56,235 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:53:56,235 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:56,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:53:56,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:53:56,236 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:56,236 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:53:56,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:56,237 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:56,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:53:56,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:56,237 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:53:56,238 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:53:56,239 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:53:56,239 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:53:56,239 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:53:56,239 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:53:56,239 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:53:56,239 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:53:56,239 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:56,240 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. started... 2023-03-29 03:53:56,240 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. as already flushing 2023-03-29 03:53:56,240 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:56,240 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:53:56,240 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:53:56,240 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:56,242 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:53:56,242 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:53:56,244 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:56,244 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:56,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:56,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:56,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:56,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:56,244 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:56,245 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:53:56,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:56,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:56,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:56,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:56,247 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:53:56,247 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@c39e05f[Count = 0] remaining members to acquire global barrier 2023-03-29 03:53:56,247 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:53:56,247 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:56,250 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:56,250 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:56,250 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:53:56,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:56,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:56,254 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:53:56,254 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:53:56,254 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:53:56,256 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:53:56,256 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:53:56,256 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:53:56,256 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:56,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:56,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:56,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:56,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:56,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:56,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:56,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:56,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:56,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:56,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:56,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:53:56,260 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:53:56,260 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:53:56,260 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:53:56,260 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:53:56,260 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:53:56,260 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:53:56,264 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:56,264 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:56,264 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:53:56,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:56,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:56,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:56,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:56,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:56,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:53:56,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:53:56,264 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:56,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:56,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:56,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:56,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:56,266 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:56,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:56,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:56,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:56,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:56,271 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:56,271 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:56,271 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:56,271 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:53:56,271 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:53:56,271 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:53:56,271 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:53:56,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:53:56,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:53:56,271 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:53:56,271 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:56,272 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:56,272 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:56,272 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:56,272 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:53:56,272 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:53:56,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:53:56,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:53:56,273 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:53:56,372 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:53:56,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:53:56,375 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:53:56,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:53:56,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:53:56,376 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:53:56,376 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:53:56,376 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:53:56,376 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:53:56,376 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:56,376 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:56,378 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:53:56,378 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:56,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:53:56,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:53:56,378 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:56,378 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:53:56,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:56,379 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:56,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:53:56,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:56,379 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:53:56,379 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:53:56,379 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:53:56,379 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:53:56,380 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:53:56,380 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:53:56,380 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:53:56,380 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:53:56,380 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:56,380 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. started... 2023-03-29 03:53:56,380 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. as already flushing 2023-03-29 03:53:56,380 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:56,380 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:53:56,380 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:53:56,380 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:56,380 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:53:56,380 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:53:56,382 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:56,382 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:56,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:56,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:56,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:56,382 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:56,382 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:53:56,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:56,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:56,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:56,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:56,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:56,384 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:53:56,384 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@35edcad3[Count = 0] remaining members to acquire global barrier 2023-03-29 03:53:56,384 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:53:56,384 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:56,386 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:56,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:56,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:56,386 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:53:56,386 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:53:56,386 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:53:56,386 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:56,386 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:53:56,392 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:56,392 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:53:56,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:56,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:56,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:56,392 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:53:56,392 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:53:56,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:56,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:56,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:56,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:56,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:56,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:56,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:56,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:53:56,396 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:53:56,396 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:53:56,396 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:53:56,396 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:53:56,396 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:53:56,396 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:53:56,398 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:56,398 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:56,398 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:53:56,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:56,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:56,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:56,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:56,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:56,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:53:56,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:56,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:53:56,399 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:56,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:56,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:56,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:56,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:56,401 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:56,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:56,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:56,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:56,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:56,409 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:56,409 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:56,409 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:56,409 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:56,409 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:56,409 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:56,409 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:56,409 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:53:56,409 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:53:56,410 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:53:56,409 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:53:56,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:53:56,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:53:56,410 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:53:56,410 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:53:56,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:53:56,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:53:56,410 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:53:56,410 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:53:56,464 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:56,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48182 deadline: 1680062096463, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:56,469 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:56,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48198 deadline: 1680062096464, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:56,510 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:53:56,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:53:56,513 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:53:56,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:53:56,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:53:56,514 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:53:56,515 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:53:56,515 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:53:56,515 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:53:56,515 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:56,515 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:56,517 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:56,517 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:53:56,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:53:56,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:53:56,518 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:56,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:56,518 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:53:56,518 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:56,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:53:56,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:56,519 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:53:56,522 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:53:56,522 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:53:56,522 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:53:56,522 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:53:56,523 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:53:56,523 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:53:56,524 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:56,524 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. started... 2023-03-29 03:53:56,524 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. as already flushing 2023-03-29 03:53:56,524 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:56,524 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:53:56,524 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:53:56,524 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:56,524 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:53:56,524 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:53:56,526 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:56,526 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:56,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:56,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:56,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:56,527 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:56,527 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:53:56,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:56,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:56,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:56,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:56,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:56,528 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:53:56,528 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1077161[Count = 0] remaining members to acquire global barrier 2023-03-29 03:53:56,528 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:53:56,529 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:56,531 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:56,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:56,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:56,531 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:53:56,531 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:53:56,531 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:53:56,531 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:56,531 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:53:56,532 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:53:56,532 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:53:56,532 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:53:56,533 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:56,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:56,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:56,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:56,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:56,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:56,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:56,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:56,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:56,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:56,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:56,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:53:56,537 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:53:56,537 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:53:56,537 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:53:56,537 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:53:56,537 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:53:56,537 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:53:56,540 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:56,540 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:53:56,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:56,540 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:56,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:56,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:56,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:56,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:56,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:56,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:53:56,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:53:56,541 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:56,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:56,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:56,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:56,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:56,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:56,543 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:56,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:56,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:56,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:56,546 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:56,546 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:53:56,546 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:56,546 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:53:56,546 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:53:56,546 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:53:56,546 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:53:56,546 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:53:56,546 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:53:56,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:53:56,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:53:56,546 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:56,547 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:56,547 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:56,547 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:56,547 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:56,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:53:56,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:53:56,646 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=601 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/C/70cfe190a5154b039755f5fbb2b6ccd5 2023-03-29 03:53:56,646 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:53:56,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:53:56,650 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:53:56,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:53:56,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:53:56,652 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:53:56,652 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:53:56,652 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:53:56,652 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:53:56,653 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/A/12d7b13eb7054aef8d0a5eb7197bcc94 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/12d7b13eb7054aef8d0a5eb7197bcc94 2023-03-29 03:53:56,654 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:56,654 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:56,656 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:53:56,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:53:56,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:53:56,656 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:56,656 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:56,656 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:53:56,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:56,657 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:56,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:53:56,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:56,657 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:53:56,658 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:53:56,658 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:53:56,658 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:53:56,658 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:53:56,659 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/12d7b13eb7054aef8d0a5eb7197bcc94, entries=250, sequenceid=601, filesize=16.7 K 2023-03-29 03:53:56,659 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:53:56,659 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:53:56,659 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:53:56,659 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:56,659 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. started... 2023-03-29 03:53:56,659 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. as already flushing 2023-03-29 03:53:56,659 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:56,659 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:53:56,659 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:53:56,659 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:56,659 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:53:56,659 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:53:56,660 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/B/530a965f47df4a2fa1336e533d825a7b as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/530a965f47df4a2fa1336e533d825a7b 2023-03-29 03:53:56,661 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:56,661 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:56,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:56,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:56,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:56,662 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:56,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:56,662 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:53:56,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:56,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:56,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:56,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:56,664 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:53:56,664 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@112b5e7c[Count = 0] remaining members to acquire global barrier 2023-03-29 03:53:56,664 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:53:56,664 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:56,665 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/530a965f47df4a2fa1336e533d825a7b, entries=150, sequenceid=601, filesize=11.9 K 2023-03-29 03:53:56,666 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:56,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:56,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:56,666 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:53:56,666 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:53:56,666 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:53:56,666 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/C/70cfe190a5154b039755f5fbb2b6ccd5 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/70cfe190a5154b039755f5fbb2b6ccd5 2023-03-29 03:53:56,666 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:56,667 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:53:56,668 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:56,668 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:53:56,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:56,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:56,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:56,668 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:53:56,668 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:53:56,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:56,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:56,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:56,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:56,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:56,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:56,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:56,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:53:56,672 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:53:56,672 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:53:56,672 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:53:56,672 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:53:56,672 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:53:56,672 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:53:56,673 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/70cfe190a5154b039755f5fbb2b6ccd5, entries=150, sequenceid=601, filesize=11.9 K 2023-03-29 03:53:56,674 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:56,674 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:53:56,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:56,674 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:56,674 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 52c064cf4695189b45c25208ecad32f7 in 986ms, sequenceid=601, compaction requested=true 2023-03-29 03:53:56,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:56,674 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 52c064cf4695189b45c25208ecad32f7: 2023-03-29 03:53:56,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:56,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:56,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:56,674 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:53:56,674 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:53:56,674 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-29 03:53:56,674 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-29 03:53:56,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:53:56,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:53:56,674 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:53:56,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:56,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:56,678 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:56,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:56,679 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61714 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-29 03:53:56,679 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 66594 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-29 03:53:56,679 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1912): 52c064cf4695189b45c25208ecad32f7/B is initiating minor compaction (all files) 2023-03-29 03:53:56,679 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1912): 52c064cf4695189b45c25208ecad32f7/A is initiating minor compaction (all files) 2023-03-29 03:53:56,679 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 52c064cf4695189b45c25208ecad32f7/B in TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:56,679 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 52c064cf4695189b45c25208ecad32f7/A in TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:56,679 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/698b56d2bd50460ba87ac4df94c10b2f, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/ec472197725345a3ab0b77e6424362ef, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/4e23e12eee9946de95509c2a1c524e95, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/7d775e277608481dae58badfd82ae313, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/530a965f47df4a2fa1336e533d825a7b] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp, totalSize=60.3 K 2023-03-29 03:53:56,679 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/90da68bdf8444aaba4f996f746916e58, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/396832ffc8ab44adb5c19073682d4869, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/0ecd46374c9747fcad6808aacf58beed, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/c0a13627cc4844f2a33f9f49d8c26510, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/12d7b13eb7054aef8d0a5eb7197bcc94] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp, totalSize=65.0 K 2023-03-29 03:53:56,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:56,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:56,680 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:56,680 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 698b56d2bd50460ba87ac4df94c10b2f, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=530, earliestPutTs=1680062033206 2023-03-29 03:53:56,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:56,680 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 90da68bdf8444aaba4f996f746916e58, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=530, earliestPutTs=1680062033206 2023-03-29 03:53:56,681 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting ec472197725345a3ab0b77e6424362ef, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=542, earliestPutTs=1680062033551 2023-03-29 03:53:56,681 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 396832ffc8ab44adb5c19073682d4869, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=542, earliestPutTs=1680062033551 2023-03-29 03:53:56,681 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 4e23e12eee9946de95509c2a1c524e95, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=581, earliestPutTs=1680062033921 2023-03-29 03:53:56,682 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 0ecd46374c9747fcad6808aacf58beed, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=581, earliestPutTs=1680062033921 2023-03-29 03:53:56,682 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 7d775e277608481dae58badfd82ae313, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=591, earliestPutTs=1680062035020 2023-03-29 03:53:56,682 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting c0a13627cc4844f2a33f9f49d8c26510, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=591, earliestPutTs=1680062035020 2023-03-29 03:53:56,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:56,683 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 530a965f47df4a2fa1336e533d825a7b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=601, earliestPutTs=1680062035648 2023-03-29 03:53:56,683 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 12d7b13eb7054aef8d0a5eb7197bcc94, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=601, earliestPutTs=1680062035648 2023-03-29 03:53:56,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:56,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:56,686 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:56,687 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:56,687 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:56,687 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:56,687 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:56,687 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:56,687 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:56,687 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:53:56,687 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:53:56,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:53:56,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:53:56,687 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:53:56,687 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:53:56,687 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:53:56,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:53:56,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:53:56,688 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:53:56,688 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:53:56,688 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:53:56,704 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] throttle.PressureAwareThroughputController(145): 52c064cf4695189b45c25208ecad32f7#A#compaction#83 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:53:56,706 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] throttle.PressureAwareThroughputController(145): 52c064cf4695189b45c25208ecad32f7#B#compaction#84 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:53:56,723 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/A/4f81282047214a51bbd1d540e4223863 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/4f81282047214a51bbd1d540e4223863 2023-03-29 03:53:56,729 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/B/ac313f6148554bea94320f08f5ccc711 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/ac313f6148554bea94320f08f5ccc711 2023-03-29 03:53:56,734 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 52c064cf4695189b45c25208ecad32f7/A of 52c064cf4695189b45c25208ecad32f7 into 4f81282047214a51bbd1d540e4223863(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:53:56,734 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 52c064cf4695189b45c25208ecad32f7: 2023-03-29 03:53:56,734 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7., storeName=52c064cf4695189b45c25208ecad32f7/A, priority=11, startTime=1680062036674; duration=0sec 2023-03-29 03:53:56,734 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:53:56,734 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-29 03:53:56,737 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61680 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-29 03:53:56,737 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1912): 52c064cf4695189b45c25208ecad32f7/C is initiating minor compaction (all files) 2023-03-29 03:53:56,737 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 52c064cf4695189b45c25208ecad32f7/C in TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:56,737 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/8a8dbebf193043dcaa8bb69122bbc562, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/00bd1303873a4740b15d08e631aac433, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/8e63a611e642470f849de454861c88db, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/05eb172415b84dd3b84da4f0797a2507, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/70cfe190a5154b039755f5fbb2b6ccd5] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp, totalSize=60.2 K 2023-03-29 03:53:56,738 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 8a8dbebf193043dcaa8bb69122bbc562, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=530, earliestPutTs=1680062033206 2023-03-29 03:53:56,738 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 00bd1303873a4740b15d08e631aac433, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=542, earliestPutTs=1680062033551 2023-03-29 03:53:56,739 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 8e63a611e642470f849de454861c88db, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=581, earliestPutTs=1680062033921 2023-03-29 03:53:56,739 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 05eb172415b84dd3b84da4f0797a2507, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=591, earliestPutTs=1680062035020 2023-03-29 03:53:56,740 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 70cfe190a5154b039755f5fbb2b6ccd5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=601, earliestPutTs=1680062035648 2023-03-29 03:53:56,745 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 52c064cf4695189b45c25208ecad32f7/B of 52c064cf4695189b45c25208ecad32f7 into ac313f6148554bea94320f08f5ccc711(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:53:56,745 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 52c064cf4695189b45c25208ecad32f7: 2023-03-29 03:53:56,745 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7., storeName=52c064cf4695189b45c25208ecad32f7/B, priority=11, startTime=1680062036674; duration=0sec 2023-03-29 03:53:56,745 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:53:56,755 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] throttle.PressureAwareThroughputController(145): 52c064cf4695189b45c25208ecad32f7#C#compaction#85 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:53:56,788 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:53:56,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:53:56,791 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:53:56,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:53:56,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:53:56,812 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:53:56,812 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:53:56,812 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:53:56,812 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:53:56,813 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:56,813 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:56,816 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:53:56,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:53:56,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:53:56,816 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:56,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:56,817 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:56,817 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:53:56,817 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:56,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:53:56,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:56,819 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:53:56,819 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/C/cf5cf78ff7364f42b52fbc05329e57d5 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/cf5cf78ff7364f42b52fbc05329e57d5 2023-03-29 03:53:56,825 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 52c064cf4695189b45c25208ecad32f7/C of 52c064cf4695189b45c25208ecad32f7 into cf5cf78ff7364f42b52fbc05329e57d5(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:53:56,825 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 52c064cf4695189b45c25208ecad32f7: 2023-03-29 03:53:56,825 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7., storeName=52c064cf4695189b45c25208ecad32f7/C, priority=11, startTime=1680062036674; duration=0sec 2023-03-29 03:53:56,825 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:53:56,828 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:53:56,828 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:53:56,828 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:53:56,828 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:53:56,828 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:53:56,836 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:53:56,836 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:53:56,836 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:56,836 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. started... 2023-03-29 03:53:56,836 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 52c064cf4695189b45c25208ecad32f7 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-03-29 03:53:56,837 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 52c064cf4695189b45c25208ecad32f7, store=A 2023-03-29 03:53:56,837 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-29 03:53:56,837 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 52c064cf4695189b45c25208ecad32f7, store=B 2023-03-29 03:53:56,837 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-29 03:53:56,837 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 52c064cf4695189b45c25208ecad32f7, store=C 2023-03-29 03:53:56,837 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-29 03:53:56,866 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=641 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/A/8f60d3565b304e83a1c3a60487aa1e63 2023-03-29 03:53:56,912 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=641 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/B/105315e182064e85af919aa4d7745385 2023-03-29 03:53:56,933 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=641 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/C/aa74d84aeca04e5a89a64b97ed2c436a 2023-03-29 03:53:56,957 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/A/8f60d3565b304e83a1c3a60487aa1e63 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/8f60d3565b304e83a1c3a60487aa1e63 2023-03-29 03:53:56,967 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/8f60d3565b304e83a1c3a60487aa1e63, entries=150, sequenceid=641, filesize=11.9 K 2023-03-29 03:53:56,968 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/B/105315e182064e85af919aa4d7745385 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/105315e182064e85af919aa4d7745385 2023-03-29 03:53:56,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on 52c064cf4695189b45c25208ecad32f7 2023-03-29 03:53:56,974 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. as already flushing 2023-03-29 03:53:56,977 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/105315e182064e85af919aa4d7745385, entries=150, sequenceid=641, filesize=11.9 K 2023-03-29 03:53:56,979 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/C/aa74d84aeca04e5a89a64b97ed2c436a as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/aa74d84aeca04e5a89a64b97ed2c436a 2023-03-29 03:53:56,986 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/aa74d84aeca04e5a89a64b97ed2c436a, entries=150, sequenceid=641, filesize=11.9 K 2023-03-29 03:53:56,987 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=40.25 KB/41220 for 52c064cf4695189b45c25208ecad32f7 in 151ms, sequenceid=641, compaction requested=false 2023-03-29 03:53:56,987 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 52c064cf4695189b45c25208ecad32f7: 2023-03-29 03:53:56,987 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:56,987 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:53:56,987 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:53:56,987 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:56,987 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:53:56,987 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:53:56,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on 52c064cf4695189b45c25208ecad32f7 2023-03-29 03:53:56,988 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 52c064cf4695189b45c25208ecad32f7 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-29 03:53:56,988 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 52c064cf4695189b45c25208ecad32f7, store=A 2023-03-29 03:53:56,988 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:53:56,988 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 52c064cf4695189b45c25208ecad32f7, store=B 2023-03-29 03:53:56,988 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:53:56,988 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 52c064cf4695189b45c25208ecad32f7, store=C 2023-03-29 03:53:56,988 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:53:56,989 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:56,989 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:56,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:56,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:56,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:56,989 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:56,990 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:53:56,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:56,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:56,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:56,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:56,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:56,993 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:53:56,993 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@52536f1e[Count = 0] remaining members to acquire global barrier 2023-03-29 03:53:56,993 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:53:56,993 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:56,996 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:56,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:56,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:56,996 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:53:56,996 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:53:56,996 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:53:56,996 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:56,996 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:53:56,998 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:53:56,998 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:53:56,998 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:53:56,998 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:56,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:56,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:56,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:56,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:56,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:56,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:57,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:57,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:57,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:57,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:57,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:53:57,002 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:53:57,002 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:53:57,002 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:53:57,002 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:53:57,002 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:53:57,002 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:53:57,006 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:57,006 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:53:57,006 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:57,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:57,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:57,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:57,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:57,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:57,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:53:57,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:57,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:53:57,007 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:57,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:57,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:57,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:57,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:57,009 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:57,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:57,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:57,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:57,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:57,017 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:53:57,017 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:53:57,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:53:57,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:53:57,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:53:57,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:53:57,018 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:57,018 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:57,018 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:57,018 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:57,018 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:57,018 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:57,018 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:53:57,018 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:57,018 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:53:57,019 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:53:57,019 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:53:57,019 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:53:57,019 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:53:57,056 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680062037017, max timestamp=1680062037054Num uniques 859; 2023-03-29 03:53:57,056 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680062037017, max timestamp=1680062037054Num uniques 859; 2023-03-29 03:53:57,056 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680062037017, max timestamp=1680062037054Num uniques 859; 2023-03-29 03:53:57,057 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-29 03:53:57,056 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-29 03:53:57,057 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-29 03:53:57,093 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:57,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 261 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48198 deadline: 1680062097092, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:57,094 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:57,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48182 deadline: 1680062097092, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:57,119 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:53:57,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:53:57,121 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:53:57,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:53:57,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:53:57,123 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:53:57,123 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:53:57,123 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:53:57,123 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:53:57,123 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:57,123 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:57,124 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:53:57,124 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:57,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:53:57,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:53:57,125 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:57,125 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:53:57,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:57,125 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:57,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:53:57,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:57,125 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:53:57,126 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:53:57,126 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:53:57,126 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:53:57,126 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:53:57,126 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:53:57,128 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:53:57,128 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:53:57,128 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:57,128 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. started... 2023-03-29 03:53:57,128 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. as already flushing 2023-03-29 03:53:57,128 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:57,128 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:53:57,128 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:53:57,128 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:57,129 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:53:57,129 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:53:57,130 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:57,130 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:57,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:57,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:57,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:57,131 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:57,131 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:53:57,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:57,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:57,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:57,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:57,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:57,132 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:53:57,132 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@9b39cd5[Count = 0] remaining members to acquire global barrier 2023-03-29 03:53:57,132 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:53:57,133 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:57,134 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:57,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:57,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:57,135 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:53:57,135 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:57,135 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:53:57,135 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:53:57,135 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:53:57,136 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:57,136 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:53:57,136 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:53:57,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:57,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:57,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:57,136 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:53:57,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:57,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:57,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:57,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:57,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:57,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:57,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:57,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:53:57,139 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:53:57,139 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:53:57,139 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:53:57,139 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:53:57,139 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:53:57,139 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:53:57,143 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:57,143 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:57,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:57,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:57,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:57,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:57,144 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:53:57,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:57,144 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:57,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:57,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:53:57,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:53:57,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:57,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:57,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:57,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:57,145 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:57,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:57,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:57,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:57,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:57,148 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:57,148 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:53:57,148 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:53:57,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:53:57,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:53:57,148 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:53:57,148 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:53:57,148 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:53:57,148 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:57,148 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:57,148 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:57,148 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:57,148 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:57,148 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:57,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:53:57,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:53:57,148 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:53:57,148 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:53:57,148 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:53:57,196 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:57,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48182 deadline: 1680062097195, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:57,196 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:57,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 263 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48198 deadline: 1680062097195, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:57,249 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:53:57,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:53:57,251 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:53:57,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:53:57,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:53:57,252 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:53:57,252 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:53:57,252 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:53:57,252 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:53:57,253 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:57,253 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:57,255 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:57,255 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:53:57,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:53:57,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:53:57,255 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:57,255 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:53:57,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:57,256 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:57,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:53:57,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:57,256 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:53:57,256 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:53:57,256 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:53:57,256 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:53:57,256 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:53:57,257 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:53:57,257 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:53:57,257 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:53:57,257 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:57,257 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. started... 2023-03-29 03:53:57,257 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. as already flushing 2023-03-29 03:53:57,257 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:57,257 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:53:57,257 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:53:57,257 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:57,257 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:53:57,257 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:53:57,259 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:57,259 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:57,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:57,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:57,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:57,259 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:57,259 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:53:57,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:57,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:57,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:57,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:57,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:57,261 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:53:57,261 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@338578fb[Count = 0] remaining members to acquire global barrier 2023-03-29 03:53:57,261 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:53:57,261 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:57,262 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:57,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:57,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:57,262 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:53:57,263 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:53:57,263 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:53:57,263 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:57,263 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:53:57,264 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:57,264 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:53:57,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:57,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:57,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:57,264 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:53:57,264 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:53:57,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:57,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:57,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:57,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:57,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:57,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:57,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:57,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:53:57,266 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:53:57,266 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:53:57,267 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:53:57,267 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:53:57,267 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:53:57,267 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:53:57,269 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:57,269 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:53:57,269 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:57,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:57,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:57,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:57,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:57,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:57,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:53:57,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:53:57,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:57,271 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:57,272 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:57,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:57,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:57,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:57,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:57,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:57,275 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:57,275 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:57,275 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:53:57,275 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:57,275 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:53:57,275 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:57,275 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:57,275 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:53:57,275 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:53:57,275 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:53:57,275 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:57,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:57,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:53:57,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:53:57,275 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:57,275 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:53:57,275 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:53:57,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:53:57,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:53:57,276 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:53:57,375 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:53:57,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:53:57,378 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:53:57,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:53:57,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:53:57,379 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:53:57,379 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:53:57,379 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:53:57,379 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:53:57,379 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:57,379 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:57,381 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:53:57,381 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:57,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:53:57,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:53:57,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:57,381 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:57,382 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:53:57,382 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:57,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:53:57,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:57,382 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:53:57,382 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:53:57,382 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:53:57,382 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:53:57,382 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:53:57,382 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:53:57,383 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:53:57,383 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:53:57,383 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:57,383 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. started... 2023-03-29 03:53:57,383 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. as already flushing 2023-03-29 03:53:57,383 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:57,383 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:53:57,383 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:53:57,383 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:57,383 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:53:57,383 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:53:57,384 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:57,384 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:57,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:57,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:57,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:57,385 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:57,385 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:53:57,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:57,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:57,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:57,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:57,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:57,386 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:53:57,386 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@78961849[Count = 0] remaining members to acquire global barrier 2023-03-29 03:53:57,386 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:53:57,386 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:57,388 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:57,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:57,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:57,388 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:53:57,388 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:53:57,388 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:53:57,388 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:57,388 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:53:57,389 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:57,389 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:53:57,390 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:53:57,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:57,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:57,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:57,390 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:53:57,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:57,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:57,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:57,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:57,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:57,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:57,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:57,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:53:57,394 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:53:57,394 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:53:57,394 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:53:57,394 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:53:57,395 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:53:57,395 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:53:57,396 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:57,396 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:57,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:57,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:57,396 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:53:57,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:57,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:57,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:57,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:53:57,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:53:57,397 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:57,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:57,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:57,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:57,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:57,398 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:57,398 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:57,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48182 deadline: 1680062097398, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:57,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:57,399 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:57,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 265 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48198 deadline: 1680062097398, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:57,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:57,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:57,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:57,405 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=651 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/A/934cac8124964286bd1bcc171c34bab6 2023-03-29 03:53:57,406 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:53:57,406 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:57,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:53:57,406 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:53:57,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:53:57,406 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:53:57,406 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:53:57,406 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:57,407 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:57,406 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:53:57,407 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:57,407 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:53:57,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:53:57,407 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:53:57,407 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:57,407 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:57,407 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:57,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:53:57,407 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:53:57,423 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=651 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/B/0d0fafb893e642bf82c38cfea677b97c 2023-03-29 03:53:57,507 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:53:57,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:53:57,509 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:53:57,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:53:57,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:53:57,510 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:53:57,510 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:53:57,510 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:53:57,510 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:53:57,511 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:57,511 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:57,512 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:53:57,512 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:57,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:53:57,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:53:57,512 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:57,513 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:53:57,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:57,513 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:57,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:53:57,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:57,513 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:53:57,514 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:53:57,514 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:53:57,514 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:53:57,514 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:53:57,514 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:53:57,514 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:53:57,514 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:53:57,514 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:57,514 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. started... 2023-03-29 03:53:57,514 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. as already flushing 2023-03-29 03:53:57,514 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:57,514 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:53:57,514 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:53:57,514 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:57,514 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:53:57,514 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:53:57,516 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:57,516 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:57,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:57,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:57,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:57,516 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:57,516 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:53:57,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:57,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:57,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:57,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:57,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:57,518 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:53:57,518 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@31f4b3da[Count = 0] remaining members to acquire global barrier 2023-03-29 03:53:57,518 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:53:57,518 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:57,519 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:57,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:57,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:57,519 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:53:57,519 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:53:57,519 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:53:57,519 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:57,519 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:53:57,521 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:57,521 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:53:57,522 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:53:57,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:57,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:57,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:57,522 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:53:57,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:57,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:57,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:57,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:57,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:57,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:57,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:57,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:53:57,525 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:53:57,525 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:53:57,525 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:53:57,525 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:53:57,525 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:53:57,525 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:53:57,527 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:57,527 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:57,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:57,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:57,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:57,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:57,527 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:53:57,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:57,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:53:57,528 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:57,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:57,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:53:57,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:57,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:57,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:57,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:57,529 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:57,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:57,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:57,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:57,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:57,531 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:57,531 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:57,532 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:57,532 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:53:57,532 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:53:57,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:53:57,532 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:57,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:53:57,532 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:53:57,532 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:53:57,532 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:53:57,532 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:57,532 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:57,532 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:57,532 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:53:57,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:53:57,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:53:57,532 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:53:57,632 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:53:57,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:53:57,635 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:53:57,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:53:57,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:53:57,636 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:53:57,636 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:53:57,636 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:53:57,636 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:53:57,636 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:57,636 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:57,638 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:53:57,638 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:57,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:53:57,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:53:57,638 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:57,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:57,638 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:53:57,638 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:57,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:53:57,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:57,639 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:53:57,644 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:53:57,644 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:53:57,644 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:53:57,644 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:53:57,644 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:53:57,646 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:53:57,646 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:53:57,646 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:57,646 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. started... 2023-03-29 03:53:57,646 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. as already flushing 2023-03-29 03:53:57,646 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:57,646 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:53:57,646 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:53:57,646 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:57,646 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:53:57,646 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:53:57,652 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:57,652 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:57,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:57,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:57,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:57,652 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:57,652 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:53:57,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:57,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:57,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:57,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:57,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:57,654 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:53:57,654 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@24b4c405[Count = 0] remaining members to acquire global barrier 2023-03-29 03:53:57,654 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:53:57,654 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:57,656 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:57,656 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:53:57,656 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:57,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:57,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:57,656 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:53:57,656 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:53:57,656 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:53:57,658 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:53:57,658 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:53:57,658 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:57,658 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:53:57,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:57,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:57,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:57,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:57,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:57,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:57,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:57,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:57,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:57,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:57,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:53:57,660 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:53:57,660 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:53:57,660 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:53:57,660 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:53:57,660 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:53:57,660 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:53:57,662 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:57,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:57,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:57,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:57,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:57,663 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:57,663 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:57,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:57,663 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:53:57,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:57,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:53:57,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:53:57,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:57,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:57,664 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:57,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:57,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:57,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:57,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:57,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:57,671 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:57,671 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:53:57,671 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:57,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:53:57,671 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:53:57,671 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:53:57,671 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:53:57,672 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:53:57,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:53:57,671 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:57,672 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:57,672 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:57,672 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:57,672 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:57,672 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:53:57,672 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:53:57,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:53:57,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:53:57,673 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:53:57,702 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:57,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48182 deadline: 1680062097701, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:57,703 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:57,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 267 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48198 deadline: 1680062097702, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:57,772 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:53:57,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:53:57,775 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:53:57,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:53:57,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:53:57,776 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:53:57,776 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:53:57,776 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:53:57,776 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:53:57,777 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:57,777 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:57,779 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:53:57,779 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:57,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:53:57,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:53:57,779 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:57,779 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:53:57,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:57,780 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:57,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:53:57,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:57,780 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:53:57,781 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:53:57,781 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:53:57,781 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:53:57,781 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:53:57,781 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:53:57,799 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:53:57,800 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:53:57,800 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:57,800 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. started... 2023-03-29 03:53:57,800 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. as already flushing 2023-03-29 03:53:57,800 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:57,800 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:53:57,800 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:53:57,800 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:57,800 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:53:57,800 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:53:57,803 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:57,803 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:57,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:57,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:57,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:57,804 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:57,804 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:53:57,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:57,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:57,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:57,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:57,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:57,805 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:53:57,805 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3cee32c8[Count = 0] remaining members to acquire global barrier 2023-03-29 03:53:57,805 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:53:57,805 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:57,806 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:57,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:57,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:57,807 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:53:57,807 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:53:57,807 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:53:57,807 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:57,807 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:53:57,808 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:53:57,808 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:53:57,808 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:53:57,809 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:57,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:57,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:57,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:57,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:57,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:57,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:57,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:57,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:57,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:57,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:57,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:53:57,813 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:53:57,813 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:53:57,813 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:53:57,813 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:53:57,813 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:53:57,813 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:53:57,815 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:57,815 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:53:57,815 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:57,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:57,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:57,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:57,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:57,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:57,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:53:57,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:53:57,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:57,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:57,816 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:57,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:57,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:57,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:57,818 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:57,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:57,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:57,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:57,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:57,822 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:57,822 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:53:57,822 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:53:57,822 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:57,822 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:57,822 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:53:57,822 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:53:57,823 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:53:57,822 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:57,823 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:53:57,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:53:57,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:53:57,823 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:57,823 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:57,823 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:57,823 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:53:57,823 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:53:57,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:53:57,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:53:57,836 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=651 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/C/983f68013b83456da3c3ebebf3213aa9 2023-03-29 03:53:57,842 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/A/934cac8124964286bd1bcc171c34bab6 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/934cac8124964286bd1bcc171c34bab6 2023-03-29 03:53:57,853 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/934cac8124964286bd1bcc171c34bab6, entries=100, sequenceid=651, filesize=9.5 K 2023-03-29 03:53:57,855 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/B/0d0fafb893e642bf82c38cfea677b97c as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/0d0fafb893e642bf82c38cfea677b97c 2023-03-29 03:53:57,869 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/0d0fafb893e642bf82c38cfea677b97c, entries=100, sequenceid=651, filesize=9.5 K 2023-03-29 03:53:57,870 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/C/983f68013b83456da3c3ebebf3213aa9 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/983f68013b83456da3c3ebebf3213aa9 2023-03-29 03:53:57,880 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/983f68013b83456da3c3ebebf3213aa9, entries=100, sequenceid=651, filesize=9.5 K 2023-03-29 03:53:57,882 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 52c064cf4695189b45c25208ecad32f7 in 894ms, sequenceid=651, compaction requested=true 2023-03-29 03:53:57,882 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 52c064cf4695189b45c25208ecad32f7: 2023-03-29 03:53:57,882 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:53:57,882 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-29 03:53:57,882 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-29 03:53:57,882 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:53:57,883 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:53:57,884 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35068 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-29 03:53:57,884 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1912): 52c064cf4695189b45c25208ecad32f7/A is initiating minor compaction (all files) 2023-03-29 03:53:57,884 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 52c064cf4695189b45c25208ecad32f7/A in TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:57,884 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/4f81282047214a51bbd1d540e4223863, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/8f60d3565b304e83a1c3a60487aa1e63, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/934cac8124964286bd1bcc171c34bab6] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp, totalSize=34.2 K 2023-03-29 03:53:57,885 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35068 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-29 03:53:57,885 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 4f81282047214a51bbd1d540e4223863, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=601, earliestPutTs=1680062035648 2023-03-29 03:53:57,885 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1912): 52c064cf4695189b45c25208ecad32f7/B is initiating minor compaction (all files) 2023-03-29 03:53:57,885 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 52c064cf4695189b45c25208ecad32f7/B in TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:57,885 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/ac313f6148554bea94320f08f5ccc711, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/105315e182064e85af919aa4d7745385, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/0d0fafb893e642bf82c38cfea677b97c] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp, totalSize=34.2 K 2023-03-29 03:53:57,886 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 8f60d3565b304e83a1c3a60487aa1e63, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=641, earliestPutTs=1680062035826 2023-03-29 03:53:57,886 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 934cac8124964286bd1bcc171c34bab6, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=651, earliestPutTs=1680062036985 2023-03-29 03:53:57,887 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting ac313f6148554bea94320f08f5ccc711, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=601, earliestPutTs=1680062035648 2023-03-29 03:53:57,888 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 105315e182064e85af919aa4d7745385, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=641, earliestPutTs=1680062035826 2023-03-29 03:53:57,889 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 0d0fafb893e642bf82c38cfea677b97c, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=651, earliestPutTs=1680062036985 2023-03-29 03:53:57,901 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] throttle.PressureAwareThroughputController(145): 52c064cf4695189b45c25208ecad32f7#A#compaction#92 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:53:57,903 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] throttle.PressureAwareThroughputController(145): 52c064cf4695189b45c25208ecad32f7#B#compaction#93 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:53:57,923 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:53:57,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:53:57,926 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:53:57,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:53:57,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:53:57,932 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:53:57,932 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:53:57,932 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:53:57,932 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:53:57,933 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:57,933 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:57,938 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:57,938 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:53:57,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:53:57,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:53:57,940 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:57,940 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:53:57,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:57,940 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:57,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:53:57,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:57,941 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:53:57,964 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/B/f0567bfd866b47b78f4987b6fac04c11 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/f0567bfd866b47b78f4987b6fac04c11 2023-03-29 03:53:57,964 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:53:57,964 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:53:57,964 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:53:57,964 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:53:57,964 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:53:57,973 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/A/4c14b25be55f4d0ea14c9e7c409bac58 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/4c14b25be55f4d0ea14c9e7c409bac58 2023-03-29 03:53:57,973 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:53:57,973 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:53:57,973 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:57,973 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. started... 2023-03-29 03:53:57,973 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 52c064cf4695189b45c25208ecad32f7 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-03-29 03:53:57,973 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 52c064cf4695189b45c25208ecad32f7, store=A 2023-03-29 03:53:57,974 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-29 03:53:57,974 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 52c064cf4695189b45c25208ecad32f7, store=B 2023-03-29 03:53:57,974 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-29 03:53:57,974 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 52c064cf4695189b45c25208ecad32f7, store=C 2023-03-29 03:53:57,974 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-29 03:53:57,979 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 52c064cf4695189b45c25208ecad32f7/B of 52c064cf4695189b45c25208ecad32f7 into f0567bfd866b47b78f4987b6fac04c11(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:53:57,979 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 52c064cf4695189b45c25208ecad32f7: 2023-03-29 03:53:57,979 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7., storeName=52c064cf4695189b45c25208ecad32f7/B, priority=13, startTime=1680062037882; duration=0sec 2023-03-29 03:53:57,980 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:53:57,980 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-29 03:53:57,982 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35034 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-29 03:53:57,982 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1912): 52c064cf4695189b45c25208ecad32f7/C is initiating minor compaction (all files) 2023-03-29 03:53:57,982 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 52c064cf4695189b45c25208ecad32f7/C in TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:57,982 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/cf5cf78ff7364f42b52fbc05329e57d5, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/aa74d84aeca04e5a89a64b97ed2c436a, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/983f68013b83456da3c3ebebf3213aa9] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp, totalSize=34.2 K 2023-03-29 03:53:57,983 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting cf5cf78ff7364f42b52fbc05329e57d5, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=601, earliestPutTs=1680062035648 2023-03-29 03:53:57,983 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 52c064cf4695189b45c25208ecad32f7/A of 52c064cf4695189b45c25208ecad32f7 into 4c14b25be55f4d0ea14c9e7c409bac58(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:53:57,983 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 52c064cf4695189b45c25208ecad32f7: 2023-03-29 03:53:57,983 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7., storeName=52c064cf4695189b45c25208ecad32f7/A, priority=13, startTime=1680062037882; duration=0sec 2023-03-29 03:53:57,983 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:53:57,984 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting aa74d84aeca04e5a89a64b97ed2c436a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=641, earliestPutTs=1680062035826 2023-03-29 03:53:57,984 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 983f68013b83456da3c3ebebf3213aa9, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=651, earliestPutTs=1680062036985 2023-03-29 03:53:58,009 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] throttle.PressureAwareThroughputController(145): 52c064cf4695189b45c25208ecad32f7#C#compaction#95 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:53:58,011 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=688 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/A/c9351b8ff3174da2b2c104d65e202f9a 2023-03-29 03:53:58,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on 52c064cf4695189b45c25208ecad32f7 2023-03-29 03:53:58,098 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. as already flushing 2023-03-29 03:53:58,124 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:58,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48252 deadline: 1680062098121, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:58,124 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:58,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48240 deadline: 1680062098122, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:58,128 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:58,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48262 deadline: 1680062098124, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:58,205 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:58,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48182 deadline: 1680062098204, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:58,211 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:58,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 269 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48198 deadline: 1680062098208, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:58,226 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:58,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48240 deadline: 1680062098225, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:58,226 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:58,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48252 deadline: 1680062098226, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:58,231 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:58,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48262 deadline: 1680062098229, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:58,428 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:58,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48240 deadline: 1680062098427, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:58,429 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:58,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48252 deadline: 1680062098429, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:58,434 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:58,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48262 deadline: 1680062098433, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:58,439 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=688 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/B/c35eebd2c86642508b36cf5a8aa3d148 2023-03-29 03:53:58,444 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/C/94d5c73b79ea458c83259402ff270ae6 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/94d5c73b79ea458c83259402ff270ae6 2023-03-29 03:53:58,450 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 52c064cf4695189b45c25208ecad32f7/C of 52c064cf4695189b45c25208ecad32f7 into 94d5c73b79ea458c83259402ff270ae6(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:53:58,450 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 52c064cf4695189b45c25208ecad32f7: 2023-03-29 03:53:58,450 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7., storeName=52c064cf4695189b45c25208ecad32f7/C, priority=13, startTime=1680062037883; duration=0sec 2023-03-29 03:53:58,451 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:53:58,454 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=688 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/C/998dd4287c35476c916f7d249bd5a4fd 2023-03-29 03:53:58,460 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/A/c9351b8ff3174da2b2c104d65e202f9a as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/c9351b8ff3174da2b2c104d65e202f9a 2023-03-29 03:53:58,466 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/c9351b8ff3174da2b2c104d65e202f9a, entries=150, sequenceid=688, filesize=11.9 K 2023-03-29 03:53:58,467 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/B/c35eebd2c86642508b36cf5a8aa3d148 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/c35eebd2c86642508b36cf5a8aa3d148 2023-03-29 03:53:58,473 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/c35eebd2c86642508b36cf5a8aa3d148, entries=150, sequenceid=688, filesize=11.9 K 2023-03-29 03:53:58,474 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/C/998dd4287c35476c916f7d249bd5a4fd as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/998dd4287c35476c916f7d249bd5a4fd 2023-03-29 03:53:58,483 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/998dd4287c35476c916f7d249bd5a4fd, entries=150, sequenceid=688, filesize=11.9 K 2023-03-29 03:53:58,484 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for 52c064cf4695189b45c25208ecad32f7 in 511ms, sequenceid=688, compaction requested=false 2023-03-29 03:53:58,485 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 52c064cf4695189b45c25208ecad32f7: 2023-03-29 03:53:58,485 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:58,485 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:53:58,485 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:53:58,485 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:58,485 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:53:58,485 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:53:58,487 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:58,487 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:58,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:58,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:58,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:58,488 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:58,488 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:53:58,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:58,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:58,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:58,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:58,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:58,490 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:53:58,490 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@74f2214c[Count = 0] remaining members to acquire global barrier 2023-03-29 03:53:58,490 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:53:58,490 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:58,491 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:58,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:58,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:58,492 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:53:58,492 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:53:58,492 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:53:58,492 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:58,492 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:53:58,494 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:53:58,494 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:53:58,494 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:53:58,495 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:58,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:58,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:58,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:58,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:58,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:58,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:58,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:58,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:58,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:58,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:58,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:53:58,498 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:53:58,498 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:53:58,498 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:53:58,498 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:53:58,498 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:53:58,498 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:53:58,499 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:58,499 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:53:58,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:58,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:58,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:53:58,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:53:58,500 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:58,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:58,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:58,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:58,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:58,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:58,501 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:58,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:58,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:58,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:58,502 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:58,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:58,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:58,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:58,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:58,506 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:53:58,506 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:53:58,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:53:58,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:53:58,506 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:58,506 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:53:58,506 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:53:58,506 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:53:58,506 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:58,506 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:58,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:53:58,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:53:58,506 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:58,506 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:53:58,506 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:53:58,506 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:58,507 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:58,507 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:58,607 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:53:58,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:53:58,609 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:53:58,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:53:58,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:53:58,610 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:53:58,610 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:53:58,610 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:53:58,610 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:53:58,611 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:58,611 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:58,612 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:58,612 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:53:58,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:53:58,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:53:58,613 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:58,613 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:53:58,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:58,613 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:58,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:53:58,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:58,614 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:53:58,614 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:53:58,614 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:53:58,614 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:53:58,614 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:53:58,614 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:53:58,614 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:53:58,614 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:53:58,614 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:58,614 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. started... 2023-03-29 03:53:58,614 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 52c064cf4695189b45c25208ecad32f7 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-29 03:53:58,614 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 52c064cf4695189b45c25208ecad32f7, store=A 2023-03-29 03:53:58,615 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:53:58,615 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 52c064cf4695189b45c25208ecad32f7, store=B 2023-03-29 03:53:58,615 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:53:58,615 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 52c064cf4695189b45c25208ecad32f7, store=C 2023-03-29 03:53:58,615 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:53:58,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on 52c064cf4695189b45c25208ecad32f7 2023-03-29 03:53:58,733 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. as already flushing 2023-03-29 03:53:58,782 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680062038742, max timestamp=1680062038781Num uniques 859; 2023-03-29 03:53:58,782 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-29 03:53:58,784 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680062038742, max timestamp=1680062038781Num uniques 859; 2023-03-29 03:53:58,784 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-29 03:53:58,785 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680062038742, max timestamp=1680062038781Num uniques 859; 2023-03-29 03:53:58,785 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-29 03:53:58,815 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:58,815 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:58,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48252 deadline: 1680062098809, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:58,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48262 deadline: 1680062098810, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:58,817 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:58,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48240 deadline: 1680062098815, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:58,918 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:58,918 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:58,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48262 deadline: 1680062098916, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:58,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48252 deadline: 1680062098916, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:58,919 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:58,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48240 deadline: 1680062098918, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:59,023 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=701 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/A/8e2267b2e71545ac9fee563966d123b3 2023-03-29 03:53:59,034 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=701 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/B/d1d531f8d71a49b3a66806981add759e 2023-03-29 03:53:59,047 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=701 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/C/c836889c1b304821a2fdd3333ce7497e 2023-03-29 03:53:59,052 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/A/8e2267b2e71545ac9fee563966d123b3 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/8e2267b2e71545ac9fee563966d123b3 2023-03-29 03:53:59,056 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/8e2267b2e71545ac9fee563966d123b3, entries=100, sequenceid=701, filesize=9.5 K 2023-03-29 03:53:59,057 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/B/d1d531f8d71a49b3a66806981add759e as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/d1d531f8d71a49b3a66806981add759e 2023-03-29 03:53:59,062 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/d1d531f8d71a49b3a66806981add759e, entries=100, sequenceid=701, filesize=9.5 K 2023-03-29 03:53:59,064 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/C/c836889c1b304821a2fdd3333ce7497e as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/c836889c1b304821a2fdd3333ce7497e 2023-03-29 03:53:59,070 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/c836889c1b304821a2fdd3333ce7497e, entries=100, sequenceid=701, filesize=9.5 K 2023-03-29 03:53:59,071 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=131.94 KB/135110 for 52c064cf4695189b45c25208ecad32f7 in 457ms, sequenceid=701, compaction requested=true 2023-03-29 03:53:59,071 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 52c064cf4695189b45c25208ecad32f7: 2023-03-29 03:53:59,071 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:59,071 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:53:59,071 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:53:59,071 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:59,071 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:53:59,071 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:53:59,073 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:59,073 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:59,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:59,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:59,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:59,074 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:59,074 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:53:59,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:59,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:59,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:59,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:59,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:59,076 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:53:59,076 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@608c366b[Count = 0] remaining members to acquire global barrier 2023-03-29 03:53:59,076 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:53:59,076 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:59,078 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:59,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:59,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:59,078 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:53:59,078 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:53:59,078 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:59,078 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:53:59,078 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:53:59,080 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:53:59,080 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:59,080 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:53:59,080 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:53:59,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:59,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:59,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:59,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:59,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:59,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:59,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:59,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:59,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:59,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:59,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:53:59,084 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:53:59,084 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:53:59,084 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:53:59,084 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:53:59,084 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:53:59,084 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:53:59,086 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:59,086 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:53:59,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:59,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:59,086 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:59,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:59,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:59,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:59,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:53:59,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:53:59,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:59,087 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:59,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:59,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:59,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:59,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:59,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:59,088 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:59,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:59,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:59,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:59,091 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:53:59,091 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:53:59,091 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:59,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:53:59,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:53:59,091 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:53:59,091 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:53:59,091 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:59,091 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:59,091 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:59,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:53:59,091 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:53:59,091 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:53:59,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:53:59,091 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:59,092 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:53:59,092 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:59,092 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:59,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on 52c064cf4695189b45c25208ecad32f7 2023-03-29 03:53:59,124 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 52c064cf4695189b45c25208ecad32f7 3/3 column families, dataSize=152.07 KB heapSize=435.69 KB 2023-03-29 03:53:59,125 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 52c064cf4695189b45c25208ecad32f7, store=A 2023-03-29 03:53:59,125 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-29 03:53:59,125 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 52c064cf4695189b45c25208ecad32f7, store=B 2023-03-29 03:53:59,125 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-29 03:53:59,125 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 52c064cf4695189b45c25208ecad32f7, store=C 2023-03-29 03:53:59,125 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-29 03:53:59,136 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:59,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48262 deadline: 1680062099132, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:59,141 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:59,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48240 deadline: 1680062099137, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:59,141 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:59,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48252 deadline: 1680062099137, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:59,192 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:53:59,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:53:59,194 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:53:59,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:53:59,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:53:59,195 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:53:59,195 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:53:59,195 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:53:59,195 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:53:59,196 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:59,196 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:59,197 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:53:59,197 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:59,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:53:59,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:53:59,197 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:59,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:59,198 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:53:59,198 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:59,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:53:59,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:59,198 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:53:59,198 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:53:59,199 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:53:59,199 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:53:59,199 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:53:59,199 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:53:59,199 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:53:59,199 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:53:59,199 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:59,199 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. started... 2023-03-29 03:53:59,199 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. as already flushing 2023-03-29 03:53:59,199 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:59,199 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:53:59,199 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:53:59,199 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:59,199 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:53:59,199 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:53:59,201 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:59,201 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:59,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:59,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:59,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:59,201 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:59,201 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:53:59,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:59,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:59,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:59,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:59,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:59,203 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:53:59,203 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@621a5293[Count = 0] remaining members to acquire global barrier 2023-03-29 03:53:59,203 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:53:59,203 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:59,204 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:59,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:59,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:59,204 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:53:59,204 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:53:59,204 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:53:59,204 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:59,204 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:53:59,207 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:59,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:59,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:59,207 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:53:59,207 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:53:59,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:59,207 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:53:59,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:59,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:59,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:59,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:59,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:59,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:59,209 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:59,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:59,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48182 deadline: 1680062099208, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:59,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:53:59,210 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:53:59,210 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:53:59,210 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:53:59,210 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:53:59,210 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:53:59,210 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:53:59,211 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:59,211 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:59,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:59,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:59,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:59,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:59,211 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:53:59,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:59,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:59,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:53:59,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:53:59,212 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:59,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:59,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:59,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:59,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:59,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:59,213 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:59,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:59,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:59,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:59,215 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:59,215 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:53:59,215 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:59,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:53:59,215 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:59,215 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:53:59,216 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:53:59,215 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:53:59,215 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:53:59,216 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:59,216 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:53:59,216 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:53:59,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:53:59,216 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:59,216 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:59,216 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:59,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:53:59,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:53:59,219 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:59,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 271 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48198 deadline: 1680062099217, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:59,239 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:59,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48262 deadline: 1680062099237, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:59,243 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:59,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48240 deadline: 1680062099242, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:59,244 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:59,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48252 deadline: 1680062099242, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:59,316 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:53:59,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:53:59,321 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:53:59,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:53:59,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:53:59,322 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:53:59,322 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:53:59,322 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:53:59,322 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:53:59,323 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:59,323 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:59,327 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:53:59,327 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:59,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:53:59,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:53:59,328 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:59,328 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:53:59,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:59,329 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:59,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:53:59,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:59,329 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:53:59,330 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:53:59,330 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:53:59,330 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:53:59,330 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:53:59,330 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:53:59,330 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:53:59,331 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:53:59,331 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:59,331 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. started... 2023-03-29 03:53:59,331 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. as already flushing 2023-03-29 03:53:59,331 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:59,331 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:53:59,331 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:53:59,331 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:59,332 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:53:59,332 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:53:59,333 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:59,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:59,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:59,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:59,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:59,333 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:59,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:59,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:59,336 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:59,337 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:53:59,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:59,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:59,338 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:53:59,338 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2faf446a[Count = 0] remaining members to acquire global barrier 2023-03-29 03:53:59,338 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:53:59,338 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:59,343 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:59,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:59,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:59,343 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:53:59,343 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:53:59,343 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:53:59,344 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:59,344 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:53:59,346 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:53:59,346 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:53:59,346 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:59,346 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:53:59,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:59,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:59,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:59,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:59,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:59,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:59,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:59,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:59,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:59,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:59,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:53:59,350 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:53:59,350 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:53:59,350 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:53:59,350 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:53:59,350 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:53:59,350 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:53:59,351 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:59,351 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:53:59,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:59,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:59,352 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:59,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:53:59,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:53:59,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:59,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:59,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:59,352 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:59,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:59,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:59,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:59,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:59,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:59,354 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:59,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:59,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:59,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:59,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:59,356 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:53:59,357 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:53:59,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:53:59,357 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:59,357 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:59,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:53:59,357 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:59,357 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:59,357 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:59,357 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:59,357 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:59,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:53:59,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:53:59,358 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:53:59,358 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:53:59,358 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:53:59,358 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:53:59,358 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:53:59,362 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:53:59,441 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:59,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48262 deadline: 1680062099441, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:59,446 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:59,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48240 deadline: 1680062099445, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:59,446 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:59,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48252 deadline: 1680062099446, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:59,458 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:53:59,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:53:59,461 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:53:59,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:53:59,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:53:59,462 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:53:59,462 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:53:59,463 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:53:59,463 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:53:59,463 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:59,463 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:59,464 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:59,465 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:59,465 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:53:59,468 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:53:59,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:53:59,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:53:59,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:59,469 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:59,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:53:59,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:59,469 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:53:59,470 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:53:59,470 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:53:59,470 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:53:59,470 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:53:59,472 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:53:59,472 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:53:59,472 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:59,472 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. started... 2023-03-29 03:53:59,472 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. as already flushing 2023-03-29 03:53:59,472 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:59,472 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:53:59,472 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:53:59,472 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:59,472 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:53:59,472 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:53:59,474 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:59,474 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:59,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:59,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:59,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:59,474 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:59,474 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:53:59,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:59,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:59,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:59,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:59,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:59,476 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:53:59,476 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a32f892[Count = 0] remaining members to acquire global barrier 2023-03-29 03:53:59,476 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:53:59,476 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:59,477 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:59,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:59,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:59,478 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:53:59,478 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:53:59,478 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:53:59,478 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:59,478 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:53:59,480 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:59,480 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:53:59,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:59,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:59,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:59,480 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:53:59,480 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:53:59,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:59,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:59,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:59,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:59,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:59,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:59,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:59,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:53:59,483 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:53:59,483 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:53:59,483 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:53:59,483 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:53:59,483 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:53:59,483 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:53:59,485 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:59,485 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:59,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:59,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:59,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:59,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:59,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:59,485 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:53:59,486 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:59,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:53:59,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:53:59,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:59,487 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:59,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:59,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:59,489 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:59,489 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:53:59,489 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:59,489 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:59,489 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:59,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:59,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:53:59,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:53:59,489 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:59,489 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:53:59,489 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:53:59,489 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:53:59,489 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:53:59,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:59,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:53:59,489 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:59,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:53:59,490 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:59,490 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:53:59,490 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:53:59,490 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:53:59,537 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=740 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/A/04383b6e25384aeaad25a64e031220f5 2023-03-29 03:53:59,590 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:53:59,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:53:59,593 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:53:59,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:53:59,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:53:59,594 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:53:59,594 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:53:59,594 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:53:59,594 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:53:59,595 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:59,595 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:59,596 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:53:59,596 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:59,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:53:59,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:53:59,597 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:59,597 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:53:59,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:59,597 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:59,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:53:59,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:59,598 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:53:59,598 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:53:59,598 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:53:59,598 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:53:59,598 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:53:59,598 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:53:59,598 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:53:59,598 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:53:59,598 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:59,599 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. started... 2023-03-29 03:53:59,599 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. as already flushing 2023-03-29 03:53:59,599 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:59,599 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:53:59,599 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:53:59,599 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:59,599 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:53:59,599 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:53:59,601 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:59,601 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:59,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:59,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:59,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:59,602 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:59,602 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:53:59,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:59,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:59,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:59,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:59,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:59,606 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:53:59,606 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b8ca0d1[Count = 0] remaining members to acquire global barrier 2023-03-29 03:53:59,606 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:53:59,606 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:59,607 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:59,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:59,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:59,607 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:53:59,607 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:53:59,607 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:53:59,607 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:59,607 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:53:59,609 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:53:59,609 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:53:59,609 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:59,609 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:53:59,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:59,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:59,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:59,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:59,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:59,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:59,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:59,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:59,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:59,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:59,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:53:59,612 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:53:59,612 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:53:59,612 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:53:59,612 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:53:59,612 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:53:59,612 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:53:59,614 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:59,614 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:59,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:59,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:59,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:59,614 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:53:59,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:59,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:59,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:59,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:53:59,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:53:59,616 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:59,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:59,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:59,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:59,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:59,617 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:59,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:59,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:59,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:59,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:59,621 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:59,621 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:53:59,621 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:59,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:53:59,621 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:59,621 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:53:59,621 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:53:59,621 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:53:59,621 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:59,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:53:59,621 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:53:59,621 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:59,621 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:53:59,621 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:53:59,621 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:59,621 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:59,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:53:59,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:53:59,721 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:53:59,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:53:59,724 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:53:59,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:53:59,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:53:59,725 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:53:59,726 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:53:59,726 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:53:59,726 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:53:59,726 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:59,726 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:59,728 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:59,728 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:59,728 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:53:59,729 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:53:59,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:53:59,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:53:59,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:59,730 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:59,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:53:59,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:59,731 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:53:59,731 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:53:59,731 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:53:59,731 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:53:59,731 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:53:59,731 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:53:59,740 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:53:59,740 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:53:59,740 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:59,740 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. started... 2023-03-29 03:53:59,740 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. as already flushing 2023-03-29 03:53:59,740 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:59,740 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:53:59,740 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:53:59,740 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:59,740 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:53:59,740 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:53:59,742 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:59,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:59,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:59,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:59,742 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:59,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:59,743 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:59,743 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:53:59,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:59,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:59,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:59,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:59,744 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:53:59,744 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b32fdf9[Count = 0] remaining members to acquire global barrier 2023-03-29 03:53:59,744 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:53:59,744 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:59,745 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:59,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:59,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:59,746 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:53:59,746 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:53:59,746 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:53:59,746 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:59,746 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:53:59,749 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:59,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48262 deadline: 1680062099746, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:59,749 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:59,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48240 deadline: 1680062099748, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:59,751 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:59,751 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:53:59,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:59,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:59,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:59,751 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:53:59,751 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:53:59,751 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:53:59,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48252 deadline: 1680062099750, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:59,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:59,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:59,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:59,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:59,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:59,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:59,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:59,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:53:59,756 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:53:59,756 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:53:59,756 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:53:59,756 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:53:59,756 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:53:59,756 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:53:59,760 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:59,760 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:59,760 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:53:59,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:59,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:59,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:59,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:59,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:59,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:53:59,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:53:59,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:59,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:59,762 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:59,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:59,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:59,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:59,763 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:59,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:59,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:59,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:59,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:59,766 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:59,766 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:53:59,766 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:53:59,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:53:59,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:53:59,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:53:59,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:53:59,767 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:59,767 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:53:59,767 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:53:59,767 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:53:59,767 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:53:59,767 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:53:59,768 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:53:59,767 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:59,768 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:59,768 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:59,768 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:59,768 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:59,868 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:53:59,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:53:59,870 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:53:59,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:53:59,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:53:59,871 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:53:59,871 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:53:59,871 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:53:59,871 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:53:59,872 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:59,872 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:59,873 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:59,873 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:53:59,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:53:59,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:53:59,873 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:59,873 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:53:59,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:59,874 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:59,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:53:59,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:59,874 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:53:59,874 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:53:59,874 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:53:59,874 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:53:59,874 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:53:59,875 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:53:59,875 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:53:59,875 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:53:59,875 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:59,875 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. started... 2023-03-29 03:53:59,875 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. as already flushing 2023-03-29 03:53:59,875 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:53:59,875 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:53:59,875 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:53:59,875 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:59,875 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:53:59,875 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:53:59,878 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:59,878 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:59,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:59,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:59,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:59,878 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:59,878 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:53:59,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:59,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:59,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:59,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:59,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:59,883 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:53:59,883 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2eeb7ae1[Count = 0] remaining members to acquire global barrier 2023-03-29 03:53:59,883 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:53:59,883 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:59,885 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:59,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:59,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:59,885 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:53:59,885 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:53:59,885 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:53:59,887 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:53:59,887 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:53:59,887 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:53:59,888 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:59,888 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:53:59,889 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:53:59,889 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:53:59,889 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:53:59,889 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:53:59,889 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:53:59,889 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:53:59,891 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:59,891 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:59,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:59,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:59,891 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:53:59,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:59,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:53:59,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:53:59,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:53:59,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:53:59,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:53:59,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:59,892 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:59,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:59,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:53:59,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:59,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:59,896 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:59,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:53:59,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:53:59,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:59,900 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:59,900 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:59,900 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:53:59,900 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:53:59,900 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:53:59,900 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:53:59,900 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:53:59,900 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:53:59,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:53:59,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:53:59,900 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:53:59,900 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:53:59,900 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:53:59,900 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:59,900 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:53:59,900 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:53:59,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:53:59,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:53:59,959 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=740 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/B/831675536cda41c29c84e0b8532bc7ae 2023-03-29 03:54:00,000 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:54:00,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:54:00,002 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:54:00,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:54:00,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:54:00,004 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:54:00,004 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:00,004 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:54:00,004 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:54:00,004 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:00,004 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:00,006 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:00,006 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:00,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:00,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:00,006 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:00,006 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:54:00,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:00,006 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:00,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:54:00,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:00,007 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:54:00,007 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:54:00,007 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:54:00,007 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:54:00,007 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:00,007 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:54:00,007 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:54:00,007 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:54:00,007 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:54:00,007 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. started... 2023-03-29 03:54:00,007 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. as already flushing 2023-03-29 03:54:00,007 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:54:00,008 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:54:00,008 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:54:00,008 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:00,008 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:54:00,008 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:54:00,009 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:00,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:00,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:00,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:00,009 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:00,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:00,010 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:00,010 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:54:00,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:00,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:00,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:00,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:00,011 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:54:00,011 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@73c0f1c6[Count = 0] remaining members to acquire global barrier 2023-03-29 03:54:00,011 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:54:00,011 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:00,012 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:00,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:00,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:00,013 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:54:00,013 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:54:00,013 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:54:00,013 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:00,013 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:54:00,016 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:54:00,016 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:00,016 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:00,016 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:54:00,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:00,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:00,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:00,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:00,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:00,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:00,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:00,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:00,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:00,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:00,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:54:00,018 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:54:00,018 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:54:00,018 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:54:00,018 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:54:00,018 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:54:00,018 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:54:00,020 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:00,020 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:00,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:00,020 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:00,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:00,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:00,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:00,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:00,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:00,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:00,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:00,027 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:00,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:00,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:00,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:00,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:00,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:00,029 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:00,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:00,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:00,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:00,031 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:00,031 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:00,032 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:00,032 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:54:00,032 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:54:00,032 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:00,032 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:00,032 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:00,032 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:00,032 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:00,032 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:00,032 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:00,032 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:54:00,032 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:54:00,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:00,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:00,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:00,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:00,132 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:54:00,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:54:00,134 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:54:00,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:54:00,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:54:00,135 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:54:00,135 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:00,136 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:54:00,136 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:54:00,136 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:00,136 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:00,137 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:00,138 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:00,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:00,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:00,138 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:00,138 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:54:00,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:00,139 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:00,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:54:00,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:00,140 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:54:00,140 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:54:00,140 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:54:00,140 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:54:00,140 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:00,140 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:54:00,140 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:54:00,141 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:54:00,141 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:54:00,141 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. started... 2023-03-29 03:54:00,141 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. as already flushing 2023-03-29 03:54:00,141 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:54:00,141 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:54:00,141 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:54:00,141 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:00,141 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:54:00,141 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:54:00,144 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:00,144 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:00,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:00,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:00,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:00,144 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:00,144 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:54:00,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:00,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:00,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:00,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:00,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:00,146 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:54:00,146 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5de45d47[Count = 0] remaining members to acquire global barrier 2023-03-29 03:54:00,146 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:54:00,146 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:00,147 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:00,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:00,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:00,147 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:54:00,147 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:54:00,147 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:54:00,148 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:00,148 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:54:00,149 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:54:00,149 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:00,149 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:00,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:00,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:00,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:00,149 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:54:00,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:00,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:00,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:00,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:00,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:00,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:00,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:00,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:54:00,152 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:54:00,152 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:54:00,152 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:54:00,152 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:54:00,152 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:54:00,152 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:54:00,154 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:00,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:00,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:00,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:00,154 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:00,154 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:00,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:00,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:00,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:00,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:00,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:00,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:00,155 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:00,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:00,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:00,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:00,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:00,157 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:00,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:00,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:00,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:00,162 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:00,162 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:00,162 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:00,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:00,162 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:00,162 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:00,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:00,162 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:54:00,163 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:54:00,162 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:00,163 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:00,163 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:00,163 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:00,163 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:00,163 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:54:00,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:00,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:00,163 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:54:00,252 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:00,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48262 deadline: 1680062100251, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:00,253 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:00,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48240 deadline: 1680062100252, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:00,259 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:00,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48252 deadline: 1680062100255, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:00,263 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:54:00,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:54:00,265 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:54:00,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:54:00,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:54:00,266 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:54:00,266 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:00,266 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:54:00,266 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:54:00,267 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:00,267 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:00,268 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:00,268 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:00,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:00,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:00,269 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:00,269 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:54:00,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:00,269 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:00,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:54:00,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:00,269 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:54:00,270 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:54:00,270 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:54:00,270 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:54:00,270 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:00,270 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:54:00,270 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:54:00,270 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:54:00,270 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:54:00,270 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. started... 2023-03-29 03:54:00,270 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. as already flushing 2023-03-29 03:54:00,270 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:54:00,270 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:54:00,270 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:54:00,270 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:00,270 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:54:00,270 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:54:00,272 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:00,272 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:00,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:00,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:00,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:00,272 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:00,272 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:54:00,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:00,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:00,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:00,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:00,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:00,273 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:54:00,273 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:54:00,273 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3155ae37[Count = 0] remaining members to acquire global barrier 2023-03-29 03:54:00,273 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:00,274 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:00,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:00,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:00,275 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:54:00,275 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:54:00,275 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:54:00,275 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:00,275 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:54:00,277 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:00,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:00,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:00,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:00,277 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:54:00,277 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:00,277 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:54:00,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:00,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:00,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:00,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:00,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:00,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:00,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:00,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:54:00,279 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:54:00,279 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:54:00,279 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:54:00,279 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:54:00,279 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:54:00,279 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:54:00,281 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:00,281 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:00,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:00,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:00,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:00,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:00,281 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:00,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:00,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:00,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:00,282 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:00,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:00,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:00,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:00,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:00,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:00,283 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:00,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:00,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:00,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:00,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:00,285 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:00,285 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:00,285 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:00,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:00,285 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:00,285 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:54:00,285 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:54:00,285 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:00,286 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:54:00,286 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:54:00,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:00,286 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:54:00,285 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:00,286 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:00,286 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:00,286 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:00,286 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:00,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:00,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:00,371 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=740 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/C/7f5326dd379b4772ab80602b7436ef35 2023-03-29 03:54:00,376 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/A/04383b6e25384aeaad25a64e031220f5 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/04383b6e25384aeaad25a64e031220f5 2023-03-29 03:54:00,384 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/04383b6e25384aeaad25a64e031220f5, entries=150, sequenceid=740, filesize=11.9 K 2023-03-29 03:54:00,385 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/B/831675536cda41c29c84e0b8532bc7ae as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/831675536cda41c29c84e0b8532bc7ae 2023-03-29 03:54:00,386 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:54:00,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:54:00,388 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:54:00,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:54:00,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:54:00,389 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:54:00,389 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:00,389 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:54:00,389 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:54:00,390 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:00,390 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:00,391 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/831675536cda41c29c84e0b8532bc7ae, entries=150, sequenceid=740, filesize=11.9 K 2023-03-29 03:54:00,392 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/C/7f5326dd379b4772ab80602b7436ef35 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/7f5326dd379b4772ab80602b7436ef35 2023-03-29 03:54:00,392 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:00,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:00,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:00,393 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:00,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:00,393 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:00,393 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:54:00,393 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:00,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:54:00,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:00,394 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:54:00,396 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:54:00,396 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:54:00,396 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:54:00,396 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:00,396 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:54:00,397 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/7f5326dd379b4772ab80602b7436ef35, entries=150, sequenceid=740, filesize=11.9 K 2023-03-29 03:54:00,398 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~152.07 KB/155720, heapSize ~435.64 KB/446096, currentSize=33.54 KB/34350 for 52c064cf4695189b45c25208ecad32f7 in 1274ms, sequenceid=740, compaction requested=true 2023-03-29 03:54:00,398 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 52c064cf4695189b45c25208ecad32f7: 2023-03-29 03:54:00,398 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:54:00,398 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:54:00,398 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-29 03:54:00,398 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-29 03:54:00,398 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:54:00,399 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:54:00,399 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:54:00,399 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:54:00,399 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. started... 2023-03-29 03:54:00,400 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 52c064cf4695189b45c25208ecad32f7 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-03-29 03:54:00,400 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 52c064cf4695189b45c25208ecad32f7, store=A 2023-03-29 03:54:00,400 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:54:00,400 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 52c064cf4695189b45c25208ecad32f7, store=B 2023-03-29 03:54:00,400 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:54:00,400 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 52c064cf4695189b45c25208ecad32f7, store=C 2023-03-29 03:54:00,400 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:54:00,401 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47356 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-29 03:54:00,401 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1912): 52c064cf4695189b45c25208ecad32f7/B is initiating minor compaction (all files) 2023-03-29 03:54:00,401 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47356 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-29 03:54:00,401 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 52c064cf4695189b45c25208ecad32f7/B in TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:54:00,401 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1912): 52c064cf4695189b45c25208ecad32f7/A is initiating minor compaction (all files) 2023-03-29 03:54:00,401 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/f0567bfd866b47b78f4987b6fac04c11, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/c35eebd2c86642508b36cf5a8aa3d148, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/d1d531f8d71a49b3a66806981add759e, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/831675536cda41c29c84e0b8532bc7ae] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp, totalSize=46.2 K 2023-03-29 03:54:00,401 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 52c064cf4695189b45c25208ecad32f7/A in TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:54:00,401 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/4c14b25be55f4d0ea14c9e7c409bac58, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/c9351b8ff3174da2b2c104d65e202f9a, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/8e2267b2e71545ac9fee563966d123b3, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/04383b6e25384aeaad25a64e031220f5] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp, totalSize=46.2 K 2023-03-29 03:54:00,402 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 4c14b25be55f4d0ea14c9e7c409bac58, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=651, earliestPutTs=1680062035846 2023-03-29 03:54:00,402 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting f0567bfd866b47b78f4987b6fac04c11, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=651, earliestPutTs=1680062035846 2023-03-29 03:54:00,402 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting c9351b8ff3174da2b2c104d65e202f9a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=688, earliestPutTs=1680062037067 2023-03-29 03:54:00,402 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting c35eebd2c86642508b36cf5a8aa3d148, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=688, earliestPutTs=1680062037067 2023-03-29 03:54:00,403 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 8e2267b2e71545ac9fee563966d123b3, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=701, earliestPutTs=1680062038120 2023-03-29 03:54:00,403 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting d1d531f8d71a49b3a66806981add759e, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=701, earliestPutTs=1680062038120 2023-03-29 03:54:00,403 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 04383b6e25384aeaad25a64e031220f5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=740, earliestPutTs=1680062038782 2023-03-29 03:54:00,404 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 831675536cda41c29c84e0b8532bc7ae, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=740, earliestPutTs=1680062038782 2023-03-29 03:54:00,422 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] throttle.PressureAwareThroughputController(145): 52c064cf4695189b45c25208ecad32f7#A#compaction#105 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:54:00,425 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] throttle.PressureAwareThroughputController(145): 52c064cf4695189b45c25208ecad32f7#B#compaction#106 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:54:00,432 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=748 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/A/b78637fa167a4af6a20339c214df7e9b 2023-03-29 03:54:00,454 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/B/5efaa94f07a246b5bae737b398600cc3 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/5efaa94f07a246b5bae737b398600cc3 2023-03-29 03:54:00,462 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 52c064cf4695189b45c25208ecad32f7/B of 52c064cf4695189b45c25208ecad32f7 into 5efaa94f07a246b5bae737b398600cc3(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:54:00,462 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 52c064cf4695189b45c25208ecad32f7: 2023-03-29 03:54:00,462 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7., storeName=52c064cf4695189b45c25208ecad32f7/B, priority=12, startTime=1680062040398; duration=0sec 2023-03-29 03:54:00,462 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:54:00,462 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-29 03:54:00,462 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=748 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/B/1e154727cd074fc9a1bbed80986df5ab 2023-03-29 03:54:00,464 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47322 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-29 03:54:00,464 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1912): 52c064cf4695189b45c25208ecad32f7/C is initiating minor compaction (all files) 2023-03-29 03:54:00,464 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 52c064cf4695189b45c25208ecad32f7/C in TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:54:00,464 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/94d5c73b79ea458c83259402ff270ae6, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/998dd4287c35476c916f7d249bd5a4fd, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/c836889c1b304821a2fdd3333ce7497e, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/7f5326dd379b4772ab80602b7436ef35] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp, totalSize=46.2 K 2023-03-29 03:54:00,464 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 94d5c73b79ea458c83259402ff270ae6, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=651, earliestPutTs=1680062035846 2023-03-29 03:54:00,465 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 998dd4287c35476c916f7d249bd5a4fd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=688, earliestPutTs=1680062037067 2023-03-29 03:54:00,466 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting c836889c1b304821a2fdd3333ce7497e, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=701, earliestPutTs=1680062038120 2023-03-29 03:54:00,466 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 7f5326dd379b4772ab80602b7436ef35, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=740, earliestPutTs=1680062038782 2023-03-29 03:54:00,479 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] throttle.PressureAwareThroughputController(145): 52c064cf4695189b45c25208ecad32f7#C#compaction#109 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:54:00,480 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=748 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/C/dc998e8829994e0e90aa154e15c64d55 2023-03-29 03:54:00,488 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/A/b78637fa167a4af6a20339c214df7e9b as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/b78637fa167a4af6a20339c214df7e9b 2023-03-29 03:54:00,491 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/C/cbd04824c8a54425b408018fb0df02ce as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/cbd04824c8a54425b408018fb0df02ce 2023-03-29 03:54:00,493 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/b78637fa167a4af6a20339c214df7e9b, entries=150, sequenceid=748, filesize=11.9 K 2023-03-29 03:54:00,494 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/B/1e154727cd074fc9a1bbed80986df5ab as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/1e154727cd074fc9a1bbed80986df5ab 2023-03-29 03:54:00,500 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 52c064cf4695189b45c25208ecad32f7/C of 52c064cf4695189b45c25208ecad32f7 into cbd04824c8a54425b408018fb0df02ce(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:54:00,500 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/1e154727cd074fc9a1bbed80986df5ab, entries=150, sequenceid=748, filesize=11.9 K 2023-03-29 03:54:00,500 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 52c064cf4695189b45c25208ecad32f7: 2023-03-29 03:54:00,501 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7., storeName=52c064cf4695189b45c25208ecad32f7/C, priority=12, startTime=1680062040398; duration=0sec 2023-03-29 03:54:00,501 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:54:00,502 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/C/dc998e8829994e0e90aa154e15c64d55 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/dc998e8829994e0e90aa154e15c64d55 2023-03-29 03:54:00,507 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/dc998e8829994e0e90aa154e15c64d55, entries=150, sequenceid=748, filesize=11.9 K 2023-03-29 03:54:00,508 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=0 B/0 for 52c064cf4695189b45c25208ecad32f7 in 109ms, sequenceid=748, compaction requested=false 2023-03-29 03:54:00,508 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 52c064cf4695189b45c25208ecad32f7: 2023-03-29 03:54:00,508 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:54:00,508 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:54:00,508 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:54:00,509 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:00,510 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:54:00,510 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:54:00,512 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:00,512 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:00,512 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:54:00,516 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:00,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:00,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:00,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:00,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:00,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:00,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:00,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:00,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:00,518 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:54:00,519 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@449cf68e[Count = 0] remaining members to acquire global barrier 2023-03-29 03:54:00,519 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:54:00,519 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:00,520 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:00,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:00,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:00,520 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:54:00,520 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:54:00,520 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:54:00,521 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:00,521 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:54:00,523 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:00,523 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:54:00,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:00,523 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:00,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:00,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:00,523 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:54:00,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:00,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:00,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:00,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:00,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:00,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:00,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:00,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:54:00,527 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:54:00,527 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:54:00,527 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:54:00,527 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:54:00,527 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:54:00,527 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:54:00,529 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:00,529 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:00,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:00,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:00,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:00,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:00,531 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:00,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:00,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:00,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:00,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:00,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:00,532 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:00,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:00,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:00,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:00,533 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:00,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:00,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:00,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:00,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:00,535 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:00,535 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:00,535 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:00,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:00,536 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:00,536 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:00,536 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:00,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:00,536 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:00,536 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:54:00,536 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:54:00,536 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:54:00,536 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:00,536 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:00,536 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:00,536 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:54:00,536 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:54:00,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:00,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:00,636 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:54:00,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:54:00,639 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:54:00,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:54:00,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:54:00,640 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:54:00,640 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:00,640 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:54:00,640 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:54:00,640 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:00,640 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:00,646 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:00,646 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:00,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:00,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:00,647 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:00,647 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:54:00,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:00,647 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:00,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:54:00,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:00,648 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:54:00,648 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:54:00,648 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:54:00,648 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:54:00,648 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:00,648 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:54:00,648 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:54:00,648 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:54:00,648 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:54:00,649 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. started... 2023-03-29 03:54:00,649 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 52c064cf4695189b45c25208ecad32f7: 2023-03-29 03:54:00,649 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:54:00,649 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:54:00,649 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:54:00,649 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:00,649 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:54:00,649 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:54:00,651 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:00,651 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:00,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:00,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:00,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:00,651 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:00,651 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:54:00,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:00,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:00,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:00,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:00,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:00,652 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:54:00,652 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@30a6711a[Count = 0] remaining members to acquire global barrier 2023-03-29 03:54:00,652 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:54:00,653 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:00,654 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:00,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:00,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:00,654 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:54:00,654 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:54:00,654 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:54:00,654 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:00,654 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:54:00,655 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:54:00,655 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:00,655 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:54:00,656 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:00,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:00,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:00,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:00,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:00,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:00,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:00,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:00,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:00,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:00,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:00,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:54:00,659 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:54:00,659 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:54:00,659 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:54:00,659 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:54:00,659 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:54:00,659 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:54:00,662 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:00,662 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:00,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:00,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:00,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:00,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:00,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:00,664 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:00,665 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:00,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:00,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:00,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:00,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:00,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:00,667 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:00,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:00,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:00,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:00,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:00,672 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:00,672 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:00,672 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:00,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:00,672 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:54:00,672 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:54:00,672 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:00,673 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:54:00,672 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:00,673 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:00,673 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:00,673 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:00,673 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:00,672 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:00,673 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:54:00,673 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:54:00,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:00,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:00,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:00,773 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:54:00,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:54:00,776 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:54:00,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:54:00,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:54:00,777 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:54:00,777 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:00,777 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:54:00,777 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:54:00,778 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:00,778 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:00,782 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:00,782 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:00,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:00,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:00,782 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:00,783 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:54:00,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:00,783 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:00,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:54:00,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:00,783 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:54:00,784 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:54:00,784 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:54:00,784 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:54:00,784 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:00,784 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:54:00,784 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:54:00,784 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:54:00,784 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:54:00,784 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. started... 2023-03-29 03:54:00,784 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 52c064cf4695189b45c25208ecad32f7: 2023-03-29 03:54:00,784 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:54:00,784 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:54:00,785 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:54:00,785 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:00,785 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:54:00,785 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:54:00,786 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:00,786 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:00,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:00,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:00,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:00,787 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:00,787 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:54:00,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:00,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:00,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:00,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:00,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:00,789 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:54:00,789 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f571c2c[Count = 0] remaining members to acquire global barrier 2023-03-29 03:54:00,789 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:54:00,789 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:00,791 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:00,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:00,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:00,791 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:54:00,791 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:54:00,791 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:54:00,791 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:00,791 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:54:00,792 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:54:00,793 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:00,792 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:00,793 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:54:00,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:00,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:00,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:00,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:00,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:00,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:00,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:00,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:00,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:00,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:00,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:54:00,795 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:54:00,795 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:54:00,795 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:54:00,795 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:54:00,796 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:54:00,796 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:54:00,798 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:00,798 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:00,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:00,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:00,798 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:00,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:00,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:00,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:00,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:00,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:00,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:00,799 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:00,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:00,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:00,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:00,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:00,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:00,800 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:00,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:00,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:00,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:00,803 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:00,803 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:00,803 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:00,803 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:00,803 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:54:00,803 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:00,803 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:54:00,803 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:00,803 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:00,804 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:00,804 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:54:00,804 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:54:00,807 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:00,807 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:00,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:00,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:00,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:00,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:00,838 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/A/6003761c9f1649c19a726baa743bd08a as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/6003761c9f1649c19a726baa743bd08a 2023-03-29 03:54:00,843 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 52c064cf4695189b45c25208ecad32f7/A of 52c064cf4695189b45c25208ecad32f7 into 6003761c9f1649c19a726baa743bd08a(size=13.1 K), total size for store is 25.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:54:00,844 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 52c064cf4695189b45c25208ecad32f7: 2023-03-29 03:54:00,844 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7., storeName=52c064cf4695189b45c25208ecad32f7/A, priority=12, startTime=1680062040398; duration=0sec 2023-03-29 03:54:00,844 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:54:00,904 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:54:00,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:54:00,906 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:54:00,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:54:00,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:54:00,908 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:54:00,908 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:00,908 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:54:00,908 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:54:00,908 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:00,908 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:00,910 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:00,910 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:00,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:00,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:00,910 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:00,910 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:54:00,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:00,911 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:00,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:54:00,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:00,911 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:54:00,911 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:54:00,911 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:54:00,911 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:54:00,911 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:00,911 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:54:00,912 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:54:00,912 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:54:00,912 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:54:00,912 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. started... 2023-03-29 03:54:00,912 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 52c064cf4695189b45c25208ecad32f7: 2023-03-29 03:54:00,912 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:54:00,912 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:54:00,912 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:54:00,912 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:00,912 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:54:00,912 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:54:00,914 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:00,914 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:00,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:00,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:00,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:00,914 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:00,914 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:54:00,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:00,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:00,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:00,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:00,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:00,916 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:54:00,916 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@536f37e7[Count = 0] remaining members to acquire global barrier 2023-03-29 03:54:00,916 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:54:00,916 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:00,922 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:00,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:00,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:00,922 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:54:00,922 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:54:00,922 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:54:00,922 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:00,922 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:54:00,924 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:54:00,924 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:00,924 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:00,924 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:54:00,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:00,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:00,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:00,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:00,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:00,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:00,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:00,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:00,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:00,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:00,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:54:00,926 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:54:00,926 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:54:00,927 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:54:00,927 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:54:00,927 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:54:00,927 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:54:00,928 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:00,928 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:00,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:00,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:00,928 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:00,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:00,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:00,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:00,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:00,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:00,929 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:00,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:00,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:00,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:00,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:00,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:00,930 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:00,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:00,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:00,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:00,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:00,934 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:00,934 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:00,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:00,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:00,934 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:00,934 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:54:00,934 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:00,934 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:00,934 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:54:00,934 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:54:00,934 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:00,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:00,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:00,934 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:00,934 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:00,934 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:00,934 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:00,934 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:54:00,934 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:54:01,035 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:54:01,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:54:01,038 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:54:01,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:54:01,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:54:01,039 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:54:01,039 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:01,039 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:54:01,039 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:54:01,039 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:01,039 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:01,042 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:01,042 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:01,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:01,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:01,043 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:01,043 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:54:01,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:01,043 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:01,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:54:01,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:01,044 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:54:01,048 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:54:01,048 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:54:01,048 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:54:01,048 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:01,048 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:54:01,052 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:54:01,053 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:54:01,053 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:54:01,053 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. started... 2023-03-29 03:54:01,053 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 52c064cf4695189b45c25208ecad32f7: 2023-03-29 03:54:01,053 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:54:01,053 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:54:01,053 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:54:01,053 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:01,053 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:54:01,053 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:54:01,055 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:01,055 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:01,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:01,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:01,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:01,055 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:01,055 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:54:01,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:01,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:01,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:01,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:01,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:01,057 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:54:01,057 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70e48382[Count = 0] remaining members to acquire global barrier 2023-03-29 03:54:01,057 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:54:01,057 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:01,058 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:01,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:01,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:01,059 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:54:01,059 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:54:01,059 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:54:01,059 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:01,059 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:54:01,060 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:01,060 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:54:01,060 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:01,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:01,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:01,060 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:54:01,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:01,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:01,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:01,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:01,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:01,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:01,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:01,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:01,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:54:01,063 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:54:01,063 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:54:01,063 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:54:01,063 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:54:01,063 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:54:01,063 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:54:01,066 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:01,066 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:01,066 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:01,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:01,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:01,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:01,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:01,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:01,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:01,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:01,067 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:01,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:01,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:01,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:01,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:01,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:01,069 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:01,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:01,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:01,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:01,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:01,071 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:01,071 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:01,071 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:01,071 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:01,071 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:01,071 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:01,071 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:01,071 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:01,071 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:54:01,072 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:54:01,072 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:54:01,071 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:01,072 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:54:01,072 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:54:01,071 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:01,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:01,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:01,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:01,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:01,172 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:54:01,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:54:01,174 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:54:01,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:54:01,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:54:01,176 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:54:01,176 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:01,176 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:54:01,176 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:54:01,177 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:01,177 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:01,178 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:01,178 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:01,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:01,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:01,178 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:01,179 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:54:01,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:01,179 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:01,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:54:01,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:01,179 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:54:01,180 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:54:01,180 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:54:01,180 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:54:01,180 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:01,180 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:54:01,187 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:54:01,187 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:54:01,187 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:54:01,187 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. started... 2023-03-29 03:54:01,187 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 52c064cf4695189b45c25208ecad32f7: 2023-03-29 03:54:01,187 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:54:01,188 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:54:01,188 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:54:01,188 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:01,188 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:54:01,188 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:54:01,191 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:01,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:01,191 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:01,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:01,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:01,191 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:01,191 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:54:01,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:01,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:01,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:01,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:01,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:01,193 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:54:01,193 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@73c90225[Count = 0] remaining members to acquire global barrier 2023-03-29 03:54:01,193 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:54:01,193 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:01,195 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:01,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:01,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:01,195 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:54:01,195 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:54:01,195 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:54:01,195 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:01,195 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:54:01,197 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:01,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:01,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:01,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:01,197 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:54:01,197 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:01,197 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:54:01,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:01,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:01,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:01,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:01,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:01,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:01,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:01,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:54:01,201 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:54:01,201 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:54:01,201 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:54:01,201 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:54:01,201 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:54:01,201 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:54:01,203 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:01,203 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:01,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:01,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:01,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:01,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:01,203 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:01,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:01,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:01,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:01,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:01,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:01,205 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:01,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:01,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:01,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:01,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:01,206 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:01,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:01,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:01,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:01,210 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:01,211 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:01,211 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:01,211 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:54:01,211 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:54:01,211 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:01,211 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:01,211 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:01,211 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:01,211 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:01,211 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:01,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:01,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:01,211 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:01,211 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:54:01,211 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:54:01,211 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:54:01,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:01,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:01,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on 52c064cf4695189b45c25208ecad32f7 2023-03-29 03:54:01,225 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 52c064cf4695189b45c25208ecad32f7 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-29 03:54:01,226 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 52c064cf4695189b45c25208ecad32f7, store=A 2023-03-29 03:54:01,226 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:54:01,226 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 52c064cf4695189b45c25208ecad32f7, store=B 2023-03-29 03:54:01,226 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:54:01,226 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 52c064cf4695189b45c25208ecad32f7, store=C 2023-03-29 03:54:01,226 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:54:01,247 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=766 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/A/4332369ad6c24904a64bf9cdee0c85e5 2023-03-29 03:54:01,273 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680062041264, max timestamp=1680062041272Num uniques 859; 2023-03-29 03:54:01,274 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-29 03:54:01,274 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680062041264, max timestamp=1680062041272Num uniques 859; 2023-03-29 03:54:01,274 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-29 03:54:01,292 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:01,292 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:01,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 281 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48198 deadline: 1680062101286, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:01,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48252 deadline: 1680062101287, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:01,301 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:01,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48262 deadline: 1680062101292, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:01,301 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:01,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48240 deadline: 1680062101297, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:01,301 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:01,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 268 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48182 deadline: 1680062101297, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:01,312 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:54:01,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:54:01,318 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:54:01,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:54:01,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:54:01,320 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:54:01,320 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:01,320 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:54:01,320 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:54:01,321 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:01,321 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:01,322 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:01,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:01,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:01,322 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:01,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:01,324 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:01,324 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:54:01,328 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:01,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:54:01,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:01,328 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:54:01,329 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:54:01,329 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:54:01,329 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:54:01,329 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:01,329 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:54:01,330 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:54:01,330 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:54:01,330 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:54:01,330 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. started... 2023-03-29 03:54:01,330 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. as already flushing 2023-03-29 03:54:01,330 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:54:01,330 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:54:01,331 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:54:01,331 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:01,331 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:54:01,331 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:54:01,332 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:01,333 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:01,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:01,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:01,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:01,333 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:01,333 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:54:01,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:01,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:01,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:01,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:01,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:01,334 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:54:01,335 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@408c60c[Count = 0] remaining members to acquire global barrier 2023-03-29 03:54:01,335 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:54:01,335 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:01,336 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:01,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:01,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:01,336 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:54:01,336 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:54:01,336 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:54:01,336 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:01,336 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:54:01,340 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:01,340 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:54:01,340 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:01,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:01,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:01,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:01,340 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:54:01,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:01,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:01,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:01,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:01,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:01,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:01,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:01,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:54:01,346 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:54:01,346 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:54:01,346 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:54:01,346 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:54:01,346 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:54:01,346 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:54:01,348 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:01,348 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:01,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:01,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:01,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:01,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:01,348 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:01,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:01,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:01,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:01,348 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:01,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:01,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:01,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:01,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:01,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:01,350 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:01,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:01,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:01,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:01,354 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:01,354 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:01,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:01,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:01,355 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:01,355 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:54:01,355 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:54:01,355 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:01,355 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:01,355 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:54:01,355 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:54:01,355 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:01,356 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:01,356 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:01,356 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:01,356 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:01,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:01,359 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:54:01,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:01,397 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:01,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48252 deadline: 1680062101394, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:01,398 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:01,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 283 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48198 deadline: 1680062101394, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:01,403 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:01,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48262 deadline: 1680062101402, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:01,403 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:01,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 246 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48240 deadline: 1680062101402, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:01,403 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:01,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 270 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48182 deadline: 1680062101403, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:01,456 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:54:01,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:54:01,458 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:54:01,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:54:01,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:54:01,459 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:54:01,459 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:01,459 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:54:01,459 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:54:01,459 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:01,459 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:01,461 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:01,461 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:01,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:01,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:01,461 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:01,461 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:54:01,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:01,462 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:01,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:54:01,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:01,462 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:54:01,462 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:54:01,462 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:54:01,463 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:54:01,463 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:01,463 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:54:01,463 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:54:01,463 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:54:01,463 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:54:01,463 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. started... 2023-03-29 03:54:01,463 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. as already flushing 2023-03-29 03:54:01,463 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:54:01,463 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:54:01,463 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:54:01,463 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:01,463 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:54:01,463 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:54:01,466 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:01,466 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:01,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:01,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:01,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:01,466 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:01,466 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:54:01,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:01,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:01,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:01,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:01,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:01,467 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:54:01,468 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2e721965[Count = 0] remaining members to acquire global barrier 2023-03-29 03:54:01,468 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:54:01,468 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:01,469 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:01,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:01,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:01,469 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:54:01,469 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:54:01,469 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:54:01,469 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:01,469 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:54:01,470 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:01,470 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:54:01,470 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:01,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:01,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:01,470 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:54:01,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:01,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:01,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:01,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:01,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:01,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:01,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:01,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:01,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:54:01,473 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:54:01,473 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:54:01,473 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:54:01,473 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:54:01,473 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:54:01,473 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:54:01,474 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:01,474 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:01,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:01,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:01,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:01,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:01,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:01,475 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:01,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:01,476 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:01,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:01,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:01,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:01,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:01,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:01,478 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:01,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:01,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:01,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:01,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:01,481 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:01,481 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:01,481 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:01,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:01,481 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:54:01,481 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:54:01,481 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:01,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:01,481 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:01,481 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:01,482 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:54:01,482 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:01,482 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:01,482 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:54:01,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:01,482 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:54:01,482 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:01,482 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:01,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:01,582 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:54:01,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:54:01,584 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:54:01,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:54:01,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:54:01,585 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:54:01,585 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:01,585 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:54:01,585 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:54:01,586 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:01,586 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:01,587 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:01,587 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:01,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:01,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:01,587 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:01,587 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:54:01,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:01,588 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:01,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:54:01,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:01,588 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:54:01,588 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:54:01,588 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:54:01,589 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:54:01,589 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:01,589 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:54:01,589 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:54:01,589 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:54:01,589 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:54:01,589 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. started... 2023-03-29 03:54:01,589 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. as already flushing 2023-03-29 03:54:01,589 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:54:01,589 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:54:01,589 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:54:01,589 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:01,589 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:54:01,589 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:54:01,594 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:01,594 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:01,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:01,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:01,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:01,594 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:01,594 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:54:01,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:01,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:01,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:01,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:01,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:01,595 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:54:01,595 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ade67ba[Count = 0] remaining members to acquire global barrier 2023-03-29 03:54:01,595 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:54:01,596 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:01,598 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:01,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:01,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:01,598 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:54:01,598 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:54:01,598 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:54:01,598 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:01,598 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:54:01,600 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:54:01,600 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:01,600 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:01,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:01,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:01,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:01,600 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:54:01,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:01,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:01,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:01,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:01,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:01,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:01,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:01,602 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:01,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:54:01,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 285 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48198 deadline: 1680062101599, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:01,603 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:54:01,603 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:54:01,603 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:54:01,603 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:54:01,603 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:54:01,603 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:54:01,603 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:01,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48252 deadline: 1680062101600, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:01,605 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:01,605 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:01,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:01,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:01,605 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:01,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:01,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:01,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:01,605 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:01,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 272 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48182 deadline: 1680062101604, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:01,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:01,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:01,606 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:01,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 248 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48240 deadline: 1680062101605, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:01,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:01,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:01,606 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:01,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:01,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:01,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:01,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:01,607 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:01,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:01,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:01,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:01,609 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:01,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48262 deadline: 1680062101606, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:01,610 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:01,610 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:01,610 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:01,610 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:01,610 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:54:01,610 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:54:01,610 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:01,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:01,610 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:01,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:01,610 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:01,610 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:01,610 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:54:01,610 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:01,610 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:54:01,610 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:01,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:01,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:01,680 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=766 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/B/a60dc2aff82241e2a9502afefd25f7f4 2023-03-29 03:54:01,710 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:54:01,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:54:01,713 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:54:01,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:54:01,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:54:01,714 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:54:01,715 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:01,715 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:54:01,715 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:54:01,715 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:01,715 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:01,718 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:01,718 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:01,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:01,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:01,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:01,719 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:01,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:54:01,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:01,720 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:54:01,724 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:54:01,724 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:54:01,724 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:54:01,724 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:01,724 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:54:01,731 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:54:01,731 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:54:01,733 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:54:01,733 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. started... 2023-03-29 03:54:01,733 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. as already flushing 2023-03-29 03:54:01,733 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:54:01,733 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:54:01,734 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:54:01,734 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:01,734 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:54:01,734 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:54:01,734 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:01,734 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:54:01,735 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:01,735 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:01,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:01,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:01,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:01,736 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:01,736 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:54:01,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:01,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:01,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:01,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:01,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:01,739 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:54:01,739 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@413d8aee[Count = 0] remaining members to acquire global barrier 2023-03-29 03:54:01,739 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:54:01,739 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:01,741 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:01,741 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:01,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:01,741 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:54:01,741 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:01,741 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:54:01,741 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:54:01,741 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:54:01,745 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:01,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:01,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:01,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:01,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:01,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:01,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:01,746 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:54:01,746 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:01,746 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:54:01,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:01,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:01,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:01,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:01,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:54:01,749 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:54:01,749 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:54:01,749 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:54:01,749 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:54:01,749 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:54:01,749 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:54:01,761 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:01,761 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:01,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:01,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:01,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:01,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:01,761 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:01,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:01,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:01,763 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:01,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:01,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:01,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:01,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:01,765 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:01,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:01,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:01,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:01,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:01,768 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:01,768 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:01,768 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:01,768 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:01,768 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:01,768 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:01,769 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:01,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:01,769 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:01,769 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:01,769 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:01,769 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:54:01,769 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:54:01,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:01,769 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:54:01,769 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:54:01,769 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:54:01,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:01,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:01,869 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:54:01,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:54:01,872 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:54:01,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:54:01,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:54:01,873 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:54:01,873 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:01,873 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:54:01,873 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:54:01,874 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:01,874 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:01,875 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:01,875 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:01,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:01,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:01,875 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:01,875 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:54:01,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:01,876 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:01,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:54:01,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:01,876 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:54:01,876 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:54:01,876 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:54:01,876 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:54:01,876 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:01,876 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:54:01,877 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:54:01,877 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:54:01,877 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:54:01,877 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. started... 2023-03-29 03:54:01,877 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. as already flushing 2023-03-29 03:54:01,877 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:54:01,877 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:54:01,877 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:54:01,877 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:01,877 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:54:01,877 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:54:01,879 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:01,879 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:01,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:01,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:01,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:01,879 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:01,879 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:54:01,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:01,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:01,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:01,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:01,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:01,882 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:54:01,882 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3470c63[Count = 0] remaining members to acquire global barrier 2023-03-29 03:54:01,882 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:54:01,882 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:01,884 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:01,884 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:01,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:01,885 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:54:01,885 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:54:01,885 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:54:01,885 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:01,885 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:54:01,886 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:01,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:01,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:01,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:01,886 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:54:01,886 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:01,887 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:54:01,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:01,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:01,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:01,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:01,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:01,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:01,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:01,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:54:01,889 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:54:01,889 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:54:01,889 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:54:01,889 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:54:01,889 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:54:01,889 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:54:01,891 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:01,891 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:01,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:01,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:01,891 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:01,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:01,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:01,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:01,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:01,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:01,891 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:01,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:01,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:01,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:01,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:01,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:01,893 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:01,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:01,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:01,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:01,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:01,895 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:01,895 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:01,895 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:01,895 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:01,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:01,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:01,895 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:01,895 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:54:01,895 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:54:01,895 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:01,895 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:01,895 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:54:01,895 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:01,895 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:01,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:01,895 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:01,895 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:54:01,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:01,895 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:54:01,906 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:01,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 287 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48198 deadline: 1680062101905, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:01,907 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:01,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48252 deadline: 1680062101906, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:01,908 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:01,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 250 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48240 deadline: 1680062101907, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:01,910 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:01,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 274 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48182 deadline: 1680062101907, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:01,911 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:01,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48262 deadline: 1680062101910, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:01,995 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:54:01,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:54:01,998 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:54:01,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:54:02,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:54:02,000 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:54:02,000 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:02,000 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:54:02,000 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:54:02,000 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:02,000 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:02,003 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:02,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:02,003 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:02,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:02,004 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:02,004 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:54:02,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:02,004 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:02,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:54:02,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:02,005 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:54:02,005 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:54:02,005 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:54:02,005 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:54:02,005 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:02,005 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:54:02,005 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:54:02,005 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:54:02,005 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:54:02,005 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. started... 2023-03-29 03:54:02,005 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. as already flushing 2023-03-29 03:54:02,006 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:54:02,006 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:54:02,006 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:54:02,006 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:02,006 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:54:02,006 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:54:02,007 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:02,007 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:02,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:02,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:02,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:02,008 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:02,008 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:54:02,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:02,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:02,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:02,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:02,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:02,009 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:54:02,009 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3dc0945f[Count = 0] remaining members to acquire global barrier 2023-03-29 03:54:02,009 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:54:02,009 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:02,010 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:02,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:02,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:02,011 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:54:02,011 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:54:02,011 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:02,011 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:54:02,011 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:54:02,015 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:02,016 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:54:02,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:02,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:02,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:02,016 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:02,016 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:54:02,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:02,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:02,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:02,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:02,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:02,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:02,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:02,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:54:02,018 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:54:02,019 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:54:02,018 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:54:02,019 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:54:02,019 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:54:02,019 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:54:02,020 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:02,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:02,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:02,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:02,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:02,021 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:02,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:02,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:02,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:02,022 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:02,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:02,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:02,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:02,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:02,024 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:02,024 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:02,024 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:02,024 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:02,025 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:02,025 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:02,025 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:02,025 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:02,025 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:54:02,025 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:54:02,025 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:54:02,025 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:54:02,025 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:54:02,026 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:02,026 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:02,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:02,026 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:02,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:02,026 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:54:02,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:02,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:02,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:02,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:02,102 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=766 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/C/085e330a524048c9bb7cd7e681705b40 2023-03-29 03:54:02,109 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/A/4332369ad6c24904a64bf9cdee0c85e5 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/4332369ad6c24904a64bf9cdee0c85e5 2023-03-29 03:54:02,130 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:54:02,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:54:02,133 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:54:02,134 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/4332369ad6c24904a64bf9cdee0c85e5, entries=200, sequenceid=766, filesize=14.3 K 2023-03-29 03:54:02,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:54:02,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:54:02,135 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:54:02,135 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:02,135 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:54:02,135 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:54:02,135 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:02,135 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:02,135 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/B/a60dc2aff82241e2a9502afefd25f7f4 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/a60dc2aff82241e2a9502afefd25f7f4 2023-03-29 03:54:02,137 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:02,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:02,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:02,137 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:02,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:02,137 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:02,137 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:54:02,138 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:02,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:54:02,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:02,138 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:54:02,138 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:54:02,139 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:54:02,139 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:02,139 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:54:02,141 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/a60dc2aff82241e2a9502afefd25f7f4, entries=150, sequenceid=766, filesize=11.9 K 2023-03-29 03:54:02,142 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:54:02,142 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:54:02,142 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:54:02,143 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. started... 2023-03-29 03:54:02,143 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. as already flushing 2023-03-29 03:54:02,143 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:54:02,143 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/C/085e330a524048c9bb7cd7e681705b40 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/085e330a524048c9bb7cd7e681705b40 2023-03-29 03:54:02,143 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:54:02,143 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:54:02,143 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:02,143 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:54:02,143 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:54:02,145 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:02,146 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:02,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:02,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:02,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:02,146 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:02,146 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:54:02,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:02,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:02,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:02,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:02,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:02,148 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:54:02,148 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@342ad7ea[Count = 0] remaining members to acquire global barrier 2023-03-29 03:54:02,148 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:54:02,148 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:02,149 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:02,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:02,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:02,149 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:54:02,149 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:54:02,149 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:54:02,149 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:02,149 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:54:02,150 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/085e330a524048c9bb7cd7e681705b40, entries=150, sequenceid=766, filesize=11.9 K 2023-03-29 03:54:02,151 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:54:02,151 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:02,151 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:54:02,151 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=129.71 KB/132820 for 52c064cf4695189b45c25208ecad32f7 in 926ms, sequenceid=766, compaction requested=true 2023-03-29 03:54:02,151 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:02,151 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 52c064cf4695189b45c25208ecad32f7: 2023-03-29 03:54:02,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:02,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:02,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:02,151 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:54:02,151 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-29 03:54:02,151 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-29 03:54:02,151 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:54:02,151 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:54:02,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:02,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:02,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:02,153 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37750 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-29 03:54:02,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:02,153 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1912): 52c064cf4695189b45c25208ecad32f7/B is initiating minor compaction (all files) 2023-03-29 03:54:02,153 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 40190 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-29 03:54:02,153 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 52c064cf4695189b45c25208ecad32f7/B in TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:54:02,153 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1912): 52c064cf4695189b45c25208ecad32f7/A is initiating minor compaction (all files) 2023-03-29 03:54:02,153 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/5efaa94f07a246b5bae737b398600cc3, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/1e154727cd074fc9a1bbed80986df5ab, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/a60dc2aff82241e2a9502afefd25f7f4] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp, totalSize=36.9 K 2023-03-29 03:54:02,153 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 52c064cf4695189b45c25208ecad32f7/A in TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:54:02,153 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/6003761c9f1649c19a726baa743bd08a, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/b78637fa167a4af6a20339c214df7e9b, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/4332369ad6c24904a64bf9cdee0c85e5] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp, totalSize=39.2 K 2023-03-29 03:54:02,154 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 5efaa94f07a246b5bae737b398600cc3, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=740, earliestPutTs=1680062038782 2023-03-29 03:54:02,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:02,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:02,154 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 6003761c9f1649c19a726baa743bd08a, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=740, earliestPutTs=1680062038782 2023-03-29 03:54:02,155 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 1e154727cd074fc9a1bbed80986df5ab, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=748, earliestPutTs=1680062039130 2023-03-29 03:54:02,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:02,156 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting a60dc2aff82241e2a9502afefd25f7f4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=766, earliestPutTs=1680062041218 2023-03-29 03:54:02,156 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting b78637fa167a4af6a20339c214df7e9b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=748, earliestPutTs=1680062039130 2023-03-29 03:54:02,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:54:02,157 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:54:02,157 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:54:02,157 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:54:02,157 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:54:02,157 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:54:02,157 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:54:02,157 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 4332369ad6c24904a64bf9cdee0c85e5, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=766, earliestPutTs=1680062041218 2023-03-29 03:54:02,166 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:02,166 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:02,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:02,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:02,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:02,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:02,168 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:02,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:02,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:02,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:02,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:02,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:02,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:02,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:02,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:02,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:02,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:02,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:02,173 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:02,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:02,174 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:02,177 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] throttle.PressureAwareThroughputController(145): 52c064cf4695189b45c25208ecad32f7#A#compaction#113 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:54:02,178 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:02,178 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:02,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:02,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:02,178 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:02,178 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:54:02,178 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:54:02,178 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:02,178 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:02,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:02,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:02,179 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:54:02,179 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:54:02,178 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:02,179 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:02,179 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:02,179 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:02,179 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:02,180 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] throttle.PressureAwareThroughputController(145): 52c064cf4695189b45c25208ecad32f7#B#compaction#114 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:54:02,196 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/A/8aafbc195cd54662ad876e05cc2f88e7 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/8aafbc195cd54662ad876e05cc2f88e7 2023-03-29 03:54:02,205 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 52c064cf4695189b45c25208ecad32f7/A of 52c064cf4695189b45c25208ecad32f7 into 8aafbc195cd54662ad876e05cc2f88e7(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:54:02,205 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 52c064cf4695189b45c25208ecad32f7: 2023-03-29 03:54:02,205 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7., storeName=52c064cf4695189b45c25208ecad32f7/A, priority=13, startTime=1680062042151; duration=0sec 2023-03-29 03:54:02,206 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:54:02,206 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-29 03:54:02,207 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37716 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-29 03:54:02,207 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/B/392347aa60c64425b4e09db17a4a6104 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/392347aa60c64425b4e09db17a4a6104 2023-03-29 03:54:02,208 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1912): 52c064cf4695189b45c25208ecad32f7/C is initiating minor compaction (all files) 2023-03-29 03:54:02,208 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 52c064cf4695189b45c25208ecad32f7/C in TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:54:02,208 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/cbd04824c8a54425b408018fb0df02ce, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/dc998e8829994e0e90aa154e15c64d55, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/085e330a524048c9bb7cd7e681705b40] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp, totalSize=36.8 K 2023-03-29 03:54:02,208 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting cbd04824c8a54425b408018fb0df02ce, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=740, earliestPutTs=1680062038782 2023-03-29 03:54:02,209 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting dc998e8829994e0e90aa154e15c64d55, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=748, earliestPutTs=1680062039130 2023-03-29 03:54:02,209 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 085e330a524048c9bb7cd7e681705b40, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=766, earliestPutTs=1680062041218 2023-03-29 03:54:02,215 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 52c064cf4695189b45c25208ecad32f7/B of 52c064cf4695189b45c25208ecad32f7 into 392347aa60c64425b4e09db17a4a6104(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:54:02,215 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 52c064cf4695189b45c25208ecad32f7: 2023-03-29 03:54:02,215 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7., storeName=52c064cf4695189b45c25208ecad32f7/B, priority=13, startTime=1680062042151; duration=0sec 2023-03-29 03:54:02,215 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:54:02,223 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] throttle.PressureAwareThroughputController(145): 52c064cf4695189b45c25208ecad32f7#C#compaction#115 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:54:02,279 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:54:02,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:54:02,282 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:54:02,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:54:02,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:54:02,291 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:54:02,291 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:02,291 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:54:02,291 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:54:02,291 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:02,291 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:02,293 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:02,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:02,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:02,293 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:02,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:02,294 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:02,294 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:54:02,294 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:02,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:54:02,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:02,294 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:54:02,296 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:54:02,296 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:54:02,296 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:54:02,296 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:02,296 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:54:02,308 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:54:02,308 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:54:02,320 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:54:02,320 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. started... 2023-03-29 03:54:02,320 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 52c064cf4695189b45c25208ecad32f7 3/3 column families, dataSize=129.71 KB heapSize=373.55 KB 2023-03-29 03:54:02,320 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 52c064cf4695189b45c25208ecad32f7, store=A 2023-03-29 03:54:02,321 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-29 03:54:02,321 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 52c064cf4695189b45c25208ecad32f7, store=B 2023-03-29 03:54:02,321 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-29 03:54:02,321 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 52c064cf4695189b45c25208ecad32f7, store=C 2023-03-29 03:54:02,321 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-29 03:54:02,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on 52c064cf4695189b45c25208ecad32f7 2023-03-29 03:54:02,413 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. as already flushing 2023-03-29 03:54:02,419 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:02,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 291 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48198 deadline: 1680062102416, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:02,419 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:02,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48252 deadline: 1680062102417, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:02,419 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:02,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 277 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48182 deadline: 1680062102418, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:02,422 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:02,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 254 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48240 deadline: 1680062102418, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:02,422 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:02,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48262 deadline: 1680062102419, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:02,522 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:02,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 279 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48182 deadline: 1680062102521, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:02,523 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:02,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48252 deadline: 1680062102521, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:02,523 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:02,523 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:02,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 293 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48198 deadline: 1680062102521, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:02,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 256 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48240 deadline: 1680062102523, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:02,523 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:02,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48262 deadline: 1680062102523, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:02,657 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/C/67926b042d044834a370b7f68e242a68 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/67926b042d044834a370b7f68e242a68 2023-03-29 03:54:02,662 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 52c064cf4695189b45c25208ecad32f7/C of 52c064cf4695189b45c25208ecad32f7 into 67926b042d044834a370b7f68e242a68(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:54:02,663 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 52c064cf4695189b45c25208ecad32f7: 2023-03-29 03:54:02,663 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7., storeName=52c064cf4695189b45c25208ecad32f7/C, priority=13, startTime=1680062042151; duration=0sec 2023-03-29 03:54:02,663 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:54:02,727 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:02,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 281 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48182 deadline: 1680062102724, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:02,728 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:02,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 258 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48240 deadline: 1680062102724, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:02,728 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:02,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48252 deadline: 1680062102725, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:02,728 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:02,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 295 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48198 deadline: 1680062102725, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:02,728 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:02,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48262 deadline: 1680062102725, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:02,774 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=799 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/A/e8f637ff32db41b8b52410273e78f48d 2023-03-29 03:54:02,787 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=799 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/B/c612972da0584715a4e37d137e048d1d 2023-03-29 03:54:03,031 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:03,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 283 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48182 deadline: 1680062103029, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:03,031 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:03,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48262 deadline: 1680062103029, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:03,032 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:03,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48252 deadline: 1680062103030, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:03,032 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:03,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 260 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48240 deadline: 1680062103030, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:03,033 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:03,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 297 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:48198 deadline: 1680062103031, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:03,199 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=62.62 KB at sequenceid=799 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/C/957cc2ab93574890a90b7f210ce12b6e 2023-03-29 03:54:03,204 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/A/e8f637ff32db41b8b52410273e78f48d as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/e8f637ff32db41b8b52410273e78f48d 2023-03-29 03:54:03,209 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/e8f637ff32db41b8b52410273e78f48d, entries=150, sequenceid=799, filesize=11.9 K 2023-03-29 03:54:03,211 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/B/c612972da0584715a4e37d137e048d1d as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/c612972da0584715a4e37d137e048d1d 2023-03-29 03:54:03,215 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/c612972da0584715a4e37d137e048d1d, entries=150, sequenceid=799, filesize=11.9 K 2023-03-29 03:54:03,216 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/C/957cc2ab93574890a90b7f210ce12b6e as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/957cc2ab93574890a90b7f210ce12b6e 2023-03-29 03:54:03,220 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/957cc2ab93574890a90b7f210ce12b6e, entries=150, sequenceid=799, filesize=11.9 K 2023-03-29 03:54:03,221 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~129.71 KB/132820, heapSize ~373.50 KB/382464, currentSize=53.67 KB/54960 for 52c064cf4695189b45c25208ecad32f7 in 901ms, sequenceid=799, compaction requested=false 2023-03-29 03:54:03,221 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 52c064cf4695189b45c25208ecad32f7: 2023-03-29 03:54:03,221 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:54:03,221 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:54:03,221 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:54:03,221 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:03,221 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:54:03,221 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:54:03,223 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:03,223 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:03,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:03,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:03,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:03,224 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:03,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:03,224 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:54:03,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:03,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:03,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:03,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:03,226 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:54:03,226 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@196d517b[Count = 0] remaining members to acquire global barrier 2023-03-29 03:54:03,226 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:54:03,226 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:03,227 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:03,227 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:03,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:03,227 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:54:03,227 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:54:03,227 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:54:03,227 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:03,227 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:54:03,229 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:54:03,229 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:03,229 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:54:03,229 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:03,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:03,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:03,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:03,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:03,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:03,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:03,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:03,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:03,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:03,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:03,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:54:03,232 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:54:03,232 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:54:03,232 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:54:03,232 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:54:03,232 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:54:03,232 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:54:03,234 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:03,234 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:03,234 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:03,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:03,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:03,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:03,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:03,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:03,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:03,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:03,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:03,235 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:03,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:03,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:03,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:03,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:03,237 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:03,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:03,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:03,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:03,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:03,239 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:03,239 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:03,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:03,239 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:03,239 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:03,239 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:54:03,239 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:54:03,239 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:54:03,239 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:03,239 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:03,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:03,239 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:03,239 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:54:03,239 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:03,239 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:54:03,240 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:03,240 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:03,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:03,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:03,340 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:54:03,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:54:03,342 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:54:03,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:54:03,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:54:03,343 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:54:03,343 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:03,343 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:54:03,343 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:54:03,344 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:03,344 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:03,345 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:03,345 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:03,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:03,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:03,346 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:03,346 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:54:03,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:03,347 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:03,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:54:03,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:03,347 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:54:03,347 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:54:03,348 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:54:03,348 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:54:03,348 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:03,348 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:54:03,348 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:54:03,348 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:54:03,349 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:54:03,349 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. started... 2023-03-29 03:54:03,349 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 52c064cf4695189b45c25208ecad32f7 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-29 03:54:03,349 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 52c064cf4695189b45c25208ecad32f7, store=A 2023-03-29 03:54:03,349 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:54:03,349 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 52c064cf4695189b45c25208ecad32f7, store=B 2023-03-29 03:54:03,349 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:54:03,349 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 52c064cf4695189b45c25208ecad32f7, store=C 2023-03-29 03:54:03,349 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:54:03,406 DEBUG [Thread-152] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6fef7a21 to 127.0.0.1:52103 2023-03-29 03:54:03,407 DEBUG [Thread-152] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-29 03:54:03,407 DEBUG [Thread-154] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2879e2e1 to 127.0.0.1:52103 2023-03-29 03:54:03,407 DEBUG [Thread-154] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-29 03:54:03,409 DEBUG [Thread-156] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6aad2e18 to 127.0.0.1:52103 2023-03-29 03:54:03,409 DEBUG [Thread-156] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-29 03:54:03,411 DEBUG [Thread-158] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4bd6fc59 to 127.0.0.1:52103 2023-03-29 03:54:03,411 DEBUG [Thread-158] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-29 03:54:03,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on 52c064cf4695189b45c25208ecad32f7 2023-03-29 03:54:03,533 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. as already flushing 2023-03-29 03:54:03,534 DEBUG [Thread-147] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x46192d17 to 127.0.0.1:52103 2023-03-29 03:54:03,534 DEBUG [Thread-147] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-29 03:54:03,534 DEBUG [Thread-149] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0dbb9ac1 to 127.0.0.1:52103 2023-03-29 03:54:03,534 DEBUG [Thread-149] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-29 03:54:03,535 DEBUG [Thread-145] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6038a026 to 127.0.0.1:52103 2023-03-29 03:54:03,535 DEBUG [Thread-145] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-29 03:54:03,536 DEBUG [Thread-143] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5e91d243 to 127.0.0.1:52103 2023-03-29 03:54:03,536 DEBUG [Thread-143] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-29 03:54:03,538 DEBUG [Thread-141] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x56c8e45e to 127.0.0.1:52103 2023-03-29 03:54:03,538 DEBUG [Thread-141] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-29 03:54:03,762 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=811 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/A/77dff205cf7d4e14a90dd3effe44f57b 2023-03-29 03:54:04,174 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=811 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/B/91df37d95b4d48ed92392ac2fe09d3dc 2023-03-29 03:54:04,595 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=811 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/C/2c378d2db3934e23b4b17bf4ba6cbb66 2023-03-29 03:54:04,599 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/A/77dff205cf7d4e14a90dd3effe44f57b as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/77dff205cf7d4e14a90dd3effe44f57b 2023-03-29 03:54:04,603 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/77dff205cf7d4e14a90dd3effe44f57b, entries=150, sequenceid=811, filesize=11.9 K 2023-03-29 03:54:04,604 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/B/91df37d95b4d48ed92392ac2fe09d3dc as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/91df37d95b4d48ed92392ac2fe09d3dc 2023-03-29 03:54:04,608 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/91df37d95b4d48ed92392ac2fe09d3dc, entries=150, sequenceid=811, filesize=11.9 K 2023-03-29 03:54:04,608 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/C/2c378d2db3934e23b4b17bf4ba6cbb66 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/2c378d2db3934e23b4b17bf4ba6cbb66 2023-03-29 03:54:04,612 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/2c378d2db3934e23b4b17bf4ba6cbb66, entries=150, sequenceid=811, filesize=11.9 K 2023-03-29 03:54:04,613 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=33.54 KB/34350 for 52c064cf4695189b45c25208ecad32f7 in 1264ms, sequenceid=811, compaction requested=true 2023-03-29 03:54:04,613 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 52c064cf4695189b45c25208ecad32f7: 2023-03-29 03:54:04,613 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:54:04,613 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:54:04,613 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:54:04,613 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:04,613 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:54:04,613 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:54:04,615 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:04,615 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:04,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:04,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:04,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:04,615 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:04,615 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:54:04,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:04,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:04,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:04,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:04,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:04,616 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:54:04,616 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@22f27221[Count = 0] remaining members to acquire global barrier 2023-03-29 03:54:04,616 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:54:04,617 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:04,625 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:04,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:04,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:04,625 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:54:04,625 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:54:04,625 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:54:04,625 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:04,625 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:54:04,627 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:04,627 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:54:04,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:04,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:04,627 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:04,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:04,627 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:54:04,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:04,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:04,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:04,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:04,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:04,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:04,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:04,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:54:04,629 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:54:04,629 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:54:04,630 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:54:04,630 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:54:04,630 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:54:04,630 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:54:04,631 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:04,631 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:04,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:04,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:04,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:04,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:04,631 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:04,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:04,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:04,632 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:04,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:04,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:04,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:04,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:04,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:04,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:04,633 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:04,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:04,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:04,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:04,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:04,635 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:04,635 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:04,635 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:04,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:04,635 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:54:04,635 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:54:04,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:04,635 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:04,635 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:04,635 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:04,635 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:04,636 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:04,636 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:54:04,636 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:04,636 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:54:04,636 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:04,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:04,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:04,736 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:54:04,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:54:04,737 INFO [Listener at localhost/35027] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-03-29 03:54:04,737 INFO [Listener at localhost/35027] hbase.AcidGuaranteesTestTool(394): wrote 167 2023-03-29 03:54:04,737 INFO [Listener at localhost/35027] hbase.AcidGuaranteesTestTool(394): wrote 125 2023-03-29 03:54:04,737 INFO [Listener at localhost/35027] hbase.AcidGuaranteesTestTool(394): wrote 151 2023-03-29 03:54:04,737 INFO [Listener at localhost/35027] hbase.AcidGuaranteesTestTool(394): wrote 113 2023-03-29 03:54:04,737 INFO [Listener at localhost/35027] hbase.AcidGuaranteesTestTool(394): wrote 132 2023-03-29 03:54:04,737 INFO [Listener at localhost/35027] hbase.AcidGuaranteesTestTool(396): Readers: 2023-03-29 03:54:04,737 INFO [Listener at localhost/35027] hbase.AcidGuaranteesTestTool(398): read 4295 2023-03-29 03:54:04,737 INFO [Listener at localhost/35027] hbase.AcidGuaranteesTestTool(398): read 4375 2023-03-29 03:54:04,737 INFO [Listener at localhost/35027] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-03-29 03:54:04,737 INFO [Listener at localhost/35027] hbase.AcidGuaranteesTestTool(402): scanned 1886 2023-03-29 03:54:04,737 INFO [Listener at localhost/35027] hbase.AcidGuaranteesTestTool(403): verified 5653 rows 2023-03-29 03:54:04,737 INFO [Listener at localhost/35027] hbase.AcidGuaranteesTestTool(402): scanned 1900 2023-03-29 03:54:04,737 INFO [Listener at localhost/35027] hbase.AcidGuaranteesTestTool(403): verified 5695 rows 2023-03-29 03:54:04,737 INFO [Listener at localhost/35027] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-03-29 03:54:04,737 DEBUG [Listener at localhost/35027] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2678d495 to 127.0.0.1:52103 2023-03-29 03:54:04,738 DEBUG [Listener at localhost/35027] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-29 03:54:04,742 INFO [Listener at localhost/35027] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-03-29 03:54:04,747 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-03-29 03:54:04,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure2.ProcedureExecutor(1029): Stored pid=12, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-03-29 03:54:04,756 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680062044756"}]},"ts":"1680062044756"} 2023-03-29 03:54:04,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-03-29 03:54:04,758 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-03-29 03:54:04,759 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-03-29 03:54:04,761 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=52c064cf4695189b45c25208ecad32f7, UNASSIGN}] 2023-03-29 03:54:04,762 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=52c064cf4695189b45c25208ecad32f7, UNASSIGN 2023-03-29 03:54:04,763 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=52c064cf4695189b45c25208ecad32f7, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:04,763 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680062044763"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680062044763"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680062044763"}]},"ts":"1680062044763"} 2023-03-29 03:54:04,765 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=14, ppid=13, state=RUNNABLE; CloseRegionProcedure 52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534}] 2023-03-29 03:54:04,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-03-29 03:54:04,923 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 52c064cf4695189b45c25208ecad32f7 2023-03-29 03:54:04,923 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 52c064cf4695189b45c25208ecad32f7, disabling compactions & flushes 2023-03-29 03:54:04,923 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:54:04,923 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:54:04,923 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. after waiting 0 ms 2023-03-29 03:54:04,923 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:54:04,923 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 52c064cf4695189b45c25208ecad32f7 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-03-29 03:54:04,923 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 52c064cf4695189b45c25208ecad32f7, store=A 2023-03-29 03:54:04,924 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:54:04,924 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 52c064cf4695189b45c25208ecad32f7, store=B 2023-03-29 03:54:04,924 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:54:04,924 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 52c064cf4695189b45c25208ecad32f7, store=C 2023-03-29 03:54:04,924 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:54:05,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-03-29 03:54:05,332 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=819 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/A/5683ee8666c54b20a0c1e8736a95f939 2023-03-29 03:54:05,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-03-29 03:54:05,743 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=819 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/B/f87086078e4d42a19bad3b42fffae9c1 2023-03-29 03:54:05,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-03-29 03:54:06,153 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=819 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/C/dba47320f85f4ef2a160d3e01e8a493c 2023-03-29 03:54:06,158 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/A/5683ee8666c54b20a0c1e8736a95f939 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/5683ee8666c54b20a0c1e8736a95f939 2023-03-29 03:54:06,162 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/5683ee8666c54b20a0c1e8736a95f939, entries=150, sequenceid=819, filesize=11.9 K 2023-03-29 03:54:06,162 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/B/f87086078e4d42a19bad3b42fffae9c1 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/f87086078e4d42a19bad3b42fffae9c1 2023-03-29 03:54:06,167 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/f87086078e4d42a19bad3b42fffae9c1, entries=150, sequenceid=819, filesize=11.9 K 2023-03-29 03:54:06,168 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/.tmp/C/dba47320f85f4ef2a160d3e01e8a493c as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/dba47320f85f4ef2a160d3e01e8a493c 2023-03-29 03:54:06,172 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/dba47320f85f4ef2a160d3e01e8a493c, entries=150, sequenceid=819, filesize=11.9 K 2023-03-29 03:54:06,173 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=0 B/0 for 52c064cf4695189b45c25208ecad32f7 in 1250ms, sequenceid=819, compaction requested=true 2023-03-29 03:54:06,199 DEBUG [StoreCloser-TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/8b45c8fe9a884504ade2b78fafe29d39, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/3310d1d5d2af4927834788258e0ea18a, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/ea13c770aa4b4756b2fb2465c8878fe4, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/03d1712dec6b499bb8ad7f3380461caa, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/d4d0ac16fd8445fe94434c01ed615900, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/c576cefc7c484428a0e43f77953a0e76, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/d620eee792fb4f95a94cfa6f8b77c04f, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/2482f46a69964531849535552193bf44, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/de45383171e342dfbb02eb813adc7f86, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/af807381d9554386bb43d0f3f86a287c, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/2acbca75782d454daf0cffd810199e73, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/70231a037c60467fb531beed1bd2a65e, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/f30a28204ca1442dbb430760c852c3c3, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/d2643fb5e294487eac330c79280e510a, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/3ae9835a527d43acb057d992b3fc1586, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/e70d5d1fcb4e4814bbcabdfda99e8f81, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/1cffb05cc65a45758c1ec05e0230a5c4, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/a61dc4c8d4f2402db89d0911daeef838, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/a2cde79c7f864bfb9e3751080a59bd22, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/e52786ef81e94ed0ba6d8fb1118466d3, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/65a875f9820e4483ba2a34c0d09006fc, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/c7fbb355302e422c90ada27b07f1d0aa, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/1914264028cf4cf786c390e2ff8fb7fd, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/90da68bdf8444aaba4f996f746916e58, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/396832ffc8ab44adb5c19073682d4869, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/0ecd46374c9747fcad6808aacf58beed, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/c0a13627cc4844f2a33f9f49d8c26510, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/12d7b13eb7054aef8d0a5eb7197bcc94, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/4f81282047214a51bbd1d540e4223863, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/8f60d3565b304e83a1c3a60487aa1e63, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/4c14b25be55f4d0ea14c9e7c409bac58, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/934cac8124964286bd1bcc171c34bab6, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/c9351b8ff3174da2b2c104d65e202f9a, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/8e2267b2e71545ac9fee563966d123b3, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/6003761c9f1649c19a726baa743bd08a, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/04383b6e25384aeaad25a64e031220f5, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/b78637fa167a4af6a20339c214df7e9b, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/4332369ad6c24904a64bf9cdee0c85e5] to archive 2023-03-29 03:54:06,201 DEBUG [StoreCloser-TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-29 03:54:06,206 DEBUG [StoreCloser-TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/8b45c8fe9a884504ade2b78fafe29d39 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/8b45c8fe9a884504ade2b78fafe29d39 2023-03-29 03:54:06,208 DEBUG [StoreCloser-TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/3310d1d5d2af4927834788258e0ea18a to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/3310d1d5d2af4927834788258e0ea18a 2023-03-29 03:54:06,210 DEBUG [StoreCloser-TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/ea13c770aa4b4756b2fb2465c8878fe4 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/ea13c770aa4b4756b2fb2465c8878fe4 2023-03-29 03:54:06,211 DEBUG [StoreCloser-TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/03d1712dec6b499bb8ad7f3380461caa to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/03d1712dec6b499bb8ad7f3380461caa 2023-03-29 03:54:06,212 DEBUG [StoreCloser-TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/d4d0ac16fd8445fe94434c01ed615900 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/d4d0ac16fd8445fe94434c01ed615900 2023-03-29 03:54:06,214 DEBUG [StoreCloser-TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/c576cefc7c484428a0e43f77953a0e76 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/c576cefc7c484428a0e43f77953a0e76 2023-03-29 03:54:06,215 DEBUG [StoreCloser-TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/d620eee792fb4f95a94cfa6f8b77c04f to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/d620eee792fb4f95a94cfa6f8b77c04f 2023-03-29 03:54:06,217 DEBUG [StoreCloser-TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/2482f46a69964531849535552193bf44 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/2482f46a69964531849535552193bf44 2023-03-29 03:54:06,218 DEBUG [StoreCloser-TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/de45383171e342dfbb02eb813adc7f86 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/de45383171e342dfbb02eb813adc7f86 2023-03-29 03:54:06,220 DEBUG [StoreCloser-TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/af807381d9554386bb43d0f3f86a287c to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/af807381d9554386bb43d0f3f86a287c 2023-03-29 03:54:06,221 DEBUG [StoreCloser-TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/2acbca75782d454daf0cffd810199e73 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/2acbca75782d454daf0cffd810199e73 2023-03-29 03:54:06,223 DEBUG [StoreCloser-TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/70231a037c60467fb531beed1bd2a65e to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/70231a037c60467fb531beed1bd2a65e 2023-03-29 03:54:06,224 DEBUG [StoreCloser-TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/f30a28204ca1442dbb430760c852c3c3 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/f30a28204ca1442dbb430760c852c3c3 2023-03-29 03:54:06,226 DEBUG [StoreCloser-TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/d2643fb5e294487eac330c79280e510a to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/d2643fb5e294487eac330c79280e510a 2023-03-29 03:54:06,227 DEBUG [StoreCloser-TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/3ae9835a527d43acb057d992b3fc1586 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/3ae9835a527d43acb057d992b3fc1586 2023-03-29 03:54:06,229 DEBUG [StoreCloser-TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/e70d5d1fcb4e4814bbcabdfda99e8f81 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/e70d5d1fcb4e4814bbcabdfda99e8f81 2023-03-29 03:54:06,230 DEBUG [StoreCloser-TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/1cffb05cc65a45758c1ec05e0230a5c4 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/1cffb05cc65a45758c1ec05e0230a5c4 2023-03-29 03:54:06,231 DEBUG [StoreCloser-TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/a61dc4c8d4f2402db89d0911daeef838 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/a61dc4c8d4f2402db89d0911daeef838 2023-03-29 03:54:06,233 DEBUG [StoreCloser-TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/a2cde79c7f864bfb9e3751080a59bd22 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/a2cde79c7f864bfb9e3751080a59bd22 2023-03-29 03:54:06,234 DEBUG [StoreCloser-TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/e52786ef81e94ed0ba6d8fb1118466d3 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/e52786ef81e94ed0ba6d8fb1118466d3 2023-03-29 03:54:06,235 DEBUG [StoreCloser-TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/65a875f9820e4483ba2a34c0d09006fc to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/65a875f9820e4483ba2a34c0d09006fc 2023-03-29 03:54:06,236 DEBUG [StoreCloser-TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/c7fbb355302e422c90ada27b07f1d0aa to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/c7fbb355302e422c90ada27b07f1d0aa 2023-03-29 03:54:06,238 DEBUG [StoreCloser-TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/1914264028cf4cf786c390e2ff8fb7fd to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/1914264028cf4cf786c390e2ff8fb7fd 2023-03-29 03:54:06,239 DEBUG [StoreCloser-TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/90da68bdf8444aaba4f996f746916e58 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/90da68bdf8444aaba4f996f746916e58 2023-03-29 03:54:06,240 DEBUG [StoreCloser-TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/396832ffc8ab44adb5c19073682d4869 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/396832ffc8ab44adb5c19073682d4869 2023-03-29 03:54:06,242 DEBUG [StoreCloser-TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/0ecd46374c9747fcad6808aacf58beed to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/0ecd46374c9747fcad6808aacf58beed 2023-03-29 03:54:06,243 DEBUG [StoreCloser-TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/c0a13627cc4844f2a33f9f49d8c26510 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/c0a13627cc4844f2a33f9f49d8c26510 2023-03-29 03:54:06,244 DEBUG [StoreCloser-TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/12d7b13eb7054aef8d0a5eb7197bcc94 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/12d7b13eb7054aef8d0a5eb7197bcc94 2023-03-29 03:54:06,245 DEBUG [StoreCloser-TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/4f81282047214a51bbd1d540e4223863 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/4f81282047214a51bbd1d540e4223863 2023-03-29 03:54:06,246 DEBUG [StoreCloser-TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/8f60d3565b304e83a1c3a60487aa1e63 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/8f60d3565b304e83a1c3a60487aa1e63 2023-03-29 03:54:06,247 DEBUG [StoreCloser-TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/4c14b25be55f4d0ea14c9e7c409bac58 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/4c14b25be55f4d0ea14c9e7c409bac58 2023-03-29 03:54:06,248 DEBUG [StoreCloser-TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/934cac8124964286bd1bcc171c34bab6 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/934cac8124964286bd1bcc171c34bab6 2023-03-29 03:54:06,249 DEBUG [StoreCloser-TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/c9351b8ff3174da2b2c104d65e202f9a to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/c9351b8ff3174da2b2c104d65e202f9a 2023-03-29 03:54:06,250 DEBUG [StoreCloser-TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/8e2267b2e71545ac9fee563966d123b3 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/8e2267b2e71545ac9fee563966d123b3 2023-03-29 03:54:06,251 DEBUG [StoreCloser-TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/6003761c9f1649c19a726baa743bd08a to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/6003761c9f1649c19a726baa743bd08a 2023-03-29 03:54:06,252 DEBUG [StoreCloser-TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/04383b6e25384aeaad25a64e031220f5 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/04383b6e25384aeaad25a64e031220f5 2023-03-29 03:54:06,253 DEBUG [StoreCloser-TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/b78637fa167a4af6a20339c214df7e9b to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/b78637fa167a4af6a20339c214df7e9b 2023-03-29 03:54:06,255 DEBUG [StoreCloser-TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/4332369ad6c24904a64bf9cdee0c85e5 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/4332369ad6c24904a64bf9cdee0c85e5 2023-03-29 03:54:06,278 DEBUG [StoreCloser-TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/ed42bd0be8ee4e5ab80efac882a336c7, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/f5b1d536c98f4340b12e0a35f5153201, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/3206fe95fbcf4b1abb047c143a5d762a, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/f514d196bbc04215b3d0f41b8fcad2db, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/e9991746cacf48aba8504b1506edde5b, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/d3052abb90da48ba928d22c36d572ac5, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/75b82637332441feb0865fefbb7a593b, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/2644b2b874074b5a890ea09af5fc0ed6, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/7de8aaa894f94c769396739b622c1c4f, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/027d19c162174e6b91142b3b7ab542f2, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/67e255c7e47d4eeeaac0edda435f2968, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/9e329bd513ce4be7ba4d19a1993c0b55, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/df4480d40c66427c971937559b5a7921, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/a9bd546764ea411d89687fb54dac8f41, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/b66c44ae8220416d8166d2cbe50a52e8, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/25b4f11b164548afab4e9ec8d417b6cb, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/96cac26339214e14b2e84fe8f7e73f28, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/1f19019b749a41ed95fd3f201f6e510e, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/7a54d5f889e9430fa4f451ce6685920e, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/597d31f73e2840b59e5d2f11937c015e, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/a3f166b8325a45a89018b251f74e6e78, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/c71cf56c2a3f4e66929198d3dd637832, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/698b56d2bd50460ba87ac4df94c10b2f, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/eb6aa41f262e45aa833562685b63e7ed, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/ec472197725345a3ab0b77e6424362ef, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/4e23e12eee9946de95509c2a1c524e95, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/7d775e277608481dae58badfd82ae313, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/ac313f6148554bea94320f08f5ccc711, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/530a965f47df4a2fa1336e533d825a7b, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/105315e182064e85af919aa4d7745385, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/f0567bfd866b47b78f4987b6fac04c11, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/0d0fafb893e642bf82c38cfea677b97c, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/c35eebd2c86642508b36cf5a8aa3d148, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/d1d531f8d71a49b3a66806981add759e, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/5efaa94f07a246b5bae737b398600cc3, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/831675536cda41c29c84e0b8532bc7ae, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/1e154727cd074fc9a1bbed80986df5ab, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/a60dc2aff82241e2a9502afefd25f7f4] to archive 2023-03-29 03:54:06,279 DEBUG [StoreCloser-TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-29 03:54:06,281 DEBUG [StoreCloser-TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/ed42bd0be8ee4e5ab80efac882a336c7 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/ed42bd0be8ee4e5ab80efac882a336c7 2023-03-29 03:54:06,282 DEBUG [StoreCloser-TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/f5b1d536c98f4340b12e0a35f5153201 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/f5b1d536c98f4340b12e0a35f5153201 2023-03-29 03:54:06,283 DEBUG [StoreCloser-TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/3206fe95fbcf4b1abb047c143a5d762a to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/3206fe95fbcf4b1abb047c143a5d762a 2023-03-29 03:54:06,284 DEBUG [StoreCloser-TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/f514d196bbc04215b3d0f41b8fcad2db to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/f514d196bbc04215b3d0f41b8fcad2db 2023-03-29 03:54:06,285 DEBUG [StoreCloser-TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/e9991746cacf48aba8504b1506edde5b to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/e9991746cacf48aba8504b1506edde5b 2023-03-29 03:54:06,286 DEBUG [StoreCloser-TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/d3052abb90da48ba928d22c36d572ac5 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/d3052abb90da48ba928d22c36d572ac5 2023-03-29 03:54:06,287 DEBUG [StoreCloser-TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/75b82637332441feb0865fefbb7a593b to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/75b82637332441feb0865fefbb7a593b 2023-03-29 03:54:06,288 DEBUG [StoreCloser-TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/2644b2b874074b5a890ea09af5fc0ed6 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/2644b2b874074b5a890ea09af5fc0ed6 2023-03-29 03:54:06,289 DEBUG [StoreCloser-TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/7de8aaa894f94c769396739b622c1c4f to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/7de8aaa894f94c769396739b622c1c4f 2023-03-29 03:54:06,290 DEBUG [StoreCloser-TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/027d19c162174e6b91142b3b7ab542f2 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/027d19c162174e6b91142b3b7ab542f2 2023-03-29 03:54:06,291 DEBUG [StoreCloser-TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/67e255c7e47d4eeeaac0edda435f2968 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/67e255c7e47d4eeeaac0edda435f2968 2023-03-29 03:54:06,292 DEBUG [StoreCloser-TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/9e329bd513ce4be7ba4d19a1993c0b55 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/9e329bd513ce4be7ba4d19a1993c0b55 2023-03-29 03:54:06,293 DEBUG [StoreCloser-TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/df4480d40c66427c971937559b5a7921 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/df4480d40c66427c971937559b5a7921 2023-03-29 03:54:06,294 DEBUG [StoreCloser-TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/a9bd546764ea411d89687fb54dac8f41 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/a9bd546764ea411d89687fb54dac8f41 2023-03-29 03:54:06,295 DEBUG [StoreCloser-TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/b66c44ae8220416d8166d2cbe50a52e8 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/b66c44ae8220416d8166d2cbe50a52e8 2023-03-29 03:54:06,296 DEBUG [StoreCloser-TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/25b4f11b164548afab4e9ec8d417b6cb to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/25b4f11b164548afab4e9ec8d417b6cb 2023-03-29 03:54:06,297 DEBUG [StoreCloser-TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/96cac26339214e14b2e84fe8f7e73f28 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/96cac26339214e14b2e84fe8f7e73f28 2023-03-29 03:54:06,298 DEBUG [StoreCloser-TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/1f19019b749a41ed95fd3f201f6e510e to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/1f19019b749a41ed95fd3f201f6e510e 2023-03-29 03:54:06,299 DEBUG [StoreCloser-TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/7a54d5f889e9430fa4f451ce6685920e to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/7a54d5f889e9430fa4f451ce6685920e 2023-03-29 03:54:06,300 DEBUG [StoreCloser-TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/597d31f73e2840b59e5d2f11937c015e to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/597d31f73e2840b59e5d2f11937c015e 2023-03-29 03:54:06,301 DEBUG [StoreCloser-TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/a3f166b8325a45a89018b251f74e6e78 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/a3f166b8325a45a89018b251f74e6e78 2023-03-29 03:54:06,303 DEBUG [StoreCloser-TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/c71cf56c2a3f4e66929198d3dd637832 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/c71cf56c2a3f4e66929198d3dd637832 2023-03-29 03:54:06,304 DEBUG [StoreCloser-TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/698b56d2bd50460ba87ac4df94c10b2f to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/698b56d2bd50460ba87ac4df94c10b2f 2023-03-29 03:54:06,305 DEBUG [StoreCloser-TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/eb6aa41f262e45aa833562685b63e7ed to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/eb6aa41f262e45aa833562685b63e7ed 2023-03-29 03:54:06,306 DEBUG [StoreCloser-TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/ec472197725345a3ab0b77e6424362ef to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/ec472197725345a3ab0b77e6424362ef 2023-03-29 03:54:06,307 DEBUG [StoreCloser-TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/4e23e12eee9946de95509c2a1c524e95 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/4e23e12eee9946de95509c2a1c524e95 2023-03-29 03:54:06,309 DEBUG [StoreCloser-TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/7d775e277608481dae58badfd82ae313 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/7d775e277608481dae58badfd82ae313 2023-03-29 03:54:06,310 DEBUG [StoreCloser-TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/ac313f6148554bea94320f08f5ccc711 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/ac313f6148554bea94320f08f5ccc711 2023-03-29 03:54:06,311 DEBUG [StoreCloser-TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/530a965f47df4a2fa1336e533d825a7b to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/530a965f47df4a2fa1336e533d825a7b 2023-03-29 03:54:06,312 DEBUG [StoreCloser-TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/105315e182064e85af919aa4d7745385 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/105315e182064e85af919aa4d7745385 2023-03-29 03:54:06,313 DEBUG [StoreCloser-TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/f0567bfd866b47b78f4987b6fac04c11 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/f0567bfd866b47b78f4987b6fac04c11 2023-03-29 03:54:06,314 DEBUG [StoreCloser-TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/0d0fafb893e642bf82c38cfea677b97c to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/0d0fafb893e642bf82c38cfea677b97c 2023-03-29 03:54:06,315 DEBUG [StoreCloser-TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/c35eebd2c86642508b36cf5a8aa3d148 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/c35eebd2c86642508b36cf5a8aa3d148 2023-03-29 03:54:06,316 DEBUG [StoreCloser-TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/d1d531f8d71a49b3a66806981add759e to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/d1d531f8d71a49b3a66806981add759e 2023-03-29 03:54:06,317 DEBUG [StoreCloser-TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/5efaa94f07a246b5bae737b398600cc3 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/5efaa94f07a246b5bae737b398600cc3 2023-03-29 03:54:06,318 DEBUG [StoreCloser-TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/831675536cda41c29c84e0b8532bc7ae to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/831675536cda41c29c84e0b8532bc7ae 2023-03-29 03:54:06,319 DEBUG [StoreCloser-TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/1e154727cd074fc9a1bbed80986df5ab to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/1e154727cd074fc9a1bbed80986df5ab 2023-03-29 03:54:06,320 DEBUG [StoreCloser-TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/a60dc2aff82241e2a9502afefd25f7f4 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/a60dc2aff82241e2a9502afefd25f7f4 2023-03-29 03:54:06,323 DEBUG [StoreCloser-TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/1cc6b52678be4b239ffa8879a819583f, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/5fdd7a7a65c6418887a869fa3d363115, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/4fe2ba50f7e8411d8ed85bfd4c2f6216, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/b817ca9e28eb4d1c87686e612bded898, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/1bf19174cb284d1db5eaac7279f1d336, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/78fe1506bd2940d7b34ef99e2f3121e0, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/a2d935a73df840608cacf3c3c925d769, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/ce816c8bd0bc442e9bb8155e8082649c, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/d62336621bae428683f69cf215e02184, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/65a9c465022245f9927e6fdbd30bad53, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/d116f3bf259446dfa2c77b7b983364e3, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/c236104cd258408eb3dd68684461d96c, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/ed5ecd8c79904eff96a4ccd90bc6ca11, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/da1dc0a233214ed8bd9a00cf08938c6d, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/8362070a6bb944dd9ac0dba88bc28ec7, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/604081a3c79341058b1dc38ce0aa3fa6, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/8c4cff2d682d45d984a884cde2fe1812, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/8bc409548e5a42f6895639d98eb01278, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/acff93129303436292586abe1900b05f, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/dffced6fd44b4df3acd08c96c4275cc0, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/edc9218aaf8c4c96880b7c1380a6d72f, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/8a8dbebf193043dcaa8bb69122bbc562, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/b58c2f5b9a924dac9a935c16f16870bf, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/00bd1303873a4740b15d08e631aac433, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/8e63a611e642470f849de454861c88db, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/05eb172415b84dd3b84da4f0797a2507, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/cf5cf78ff7364f42b52fbc05329e57d5, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/70cfe190a5154b039755f5fbb2b6ccd5, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/aa74d84aeca04e5a89a64b97ed2c436a, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/94d5c73b79ea458c83259402ff270ae6, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/983f68013b83456da3c3ebebf3213aa9, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/998dd4287c35476c916f7d249bd5a4fd, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/c836889c1b304821a2fdd3333ce7497e, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/cbd04824c8a54425b408018fb0df02ce, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/7f5326dd379b4772ab80602b7436ef35, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/dc998e8829994e0e90aa154e15c64d55, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/085e330a524048c9bb7cd7e681705b40] to archive 2023-03-29 03:54:06,324 DEBUG [StoreCloser-TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-29 03:54:06,326 DEBUG [StoreCloser-TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/1cc6b52678be4b239ffa8879a819583f to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/1cc6b52678be4b239ffa8879a819583f 2023-03-29 03:54:06,327 DEBUG [StoreCloser-TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/5fdd7a7a65c6418887a869fa3d363115 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/5fdd7a7a65c6418887a869fa3d363115 2023-03-29 03:54:06,328 DEBUG [StoreCloser-TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/4fe2ba50f7e8411d8ed85bfd4c2f6216 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/4fe2ba50f7e8411d8ed85bfd4c2f6216 2023-03-29 03:54:06,329 DEBUG [StoreCloser-TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/b817ca9e28eb4d1c87686e612bded898 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/b817ca9e28eb4d1c87686e612bded898 2023-03-29 03:54:06,330 DEBUG [StoreCloser-TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/1bf19174cb284d1db5eaac7279f1d336 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/1bf19174cb284d1db5eaac7279f1d336 2023-03-29 03:54:06,331 DEBUG [StoreCloser-TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/78fe1506bd2940d7b34ef99e2f3121e0 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/78fe1506bd2940d7b34ef99e2f3121e0 2023-03-29 03:54:06,332 DEBUG [StoreCloser-TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/a2d935a73df840608cacf3c3c925d769 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/a2d935a73df840608cacf3c3c925d769 2023-03-29 03:54:06,333 DEBUG [StoreCloser-TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/ce816c8bd0bc442e9bb8155e8082649c to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/ce816c8bd0bc442e9bb8155e8082649c 2023-03-29 03:54:06,334 DEBUG [StoreCloser-TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/d62336621bae428683f69cf215e02184 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/d62336621bae428683f69cf215e02184 2023-03-29 03:54:06,335 DEBUG [StoreCloser-TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/65a9c465022245f9927e6fdbd30bad53 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/65a9c465022245f9927e6fdbd30bad53 2023-03-29 03:54:06,337 DEBUG [StoreCloser-TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/d116f3bf259446dfa2c77b7b983364e3 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/d116f3bf259446dfa2c77b7b983364e3 2023-03-29 03:54:06,337 DEBUG [StoreCloser-TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/c236104cd258408eb3dd68684461d96c to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/c236104cd258408eb3dd68684461d96c 2023-03-29 03:54:06,338 DEBUG [StoreCloser-TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/ed5ecd8c79904eff96a4ccd90bc6ca11 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/ed5ecd8c79904eff96a4ccd90bc6ca11 2023-03-29 03:54:06,339 DEBUG [StoreCloser-TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/da1dc0a233214ed8bd9a00cf08938c6d to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/da1dc0a233214ed8bd9a00cf08938c6d 2023-03-29 03:54:06,340 DEBUG [StoreCloser-TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/8362070a6bb944dd9ac0dba88bc28ec7 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/8362070a6bb944dd9ac0dba88bc28ec7 2023-03-29 03:54:06,341 DEBUG [StoreCloser-TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/604081a3c79341058b1dc38ce0aa3fa6 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/604081a3c79341058b1dc38ce0aa3fa6 2023-03-29 03:54:06,342 DEBUG [StoreCloser-TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/8c4cff2d682d45d984a884cde2fe1812 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/8c4cff2d682d45d984a884cde2fe1812 2023-03-29 03:54:06,344 DEBUG [StoreCloser-TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/8bc409548e5a42f6895639d98eb01278 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/8bc409548e5a42f6895639d98eb01278 2023-03-29 03:54:06,345 DEBUG [StoreCloser-TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/acff93129303436292586abe1900b05f to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/acff93129303436292586abe1900b05f 2023-03-29 03:54:06,346 DEBUG [StoreCloser-TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/dffced6fd44b4df3acd08c96c4275cc0 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/dffced6fd44b4df3acd08c96c4275cc0 2023-03-29 03:54:06,347 DEBUG [StoreCloser-TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/edc9218aaf8c4c96880b7c1380a6d72f to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/edc9218aaf8c4c96880b7c1380a6d72f 2023-03-29 03:54:06,348 DEBUG [StoreCloser-TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/8a8dbebf193043dcaa8bb69122bbc562 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/8a8dbebf193043dcaa8bb69122bbc562 2023-03-29 03:54:06,350 DEBUG [StoreCloser-TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/b58c2f5b9a924dac9a935c16f16870bf to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/b58c2f5b9a924dac9a935c16f16870bf 2023-03-29 03:54:06,351 DEBUG [StoreCloser-TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/00bd1303873a4740b15d08e631aac433 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/00bd1303873a4740b15d08e631aac433 2023-03-29 03:54:06,352 DEBUG [StoreCloser-TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/8e63a611e642470f849de454861c88db to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/8e63a611e642470f849de454861c88db 2023-03-29 03:54:06,353 DEBUG [StoreCloser-TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/05eb172415b84dd3b84da4f0797a2507 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/05eb172415b84dd3b84da4f0797a2507 2023-03-29 03:54:06,355 DEBUG [StoreCloser-TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/cf5cf78ff7364f42b52fbc05329e57d5 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/cf5cf78ff7364f42b52fbc05329e57d5 2023-03-29 03:54:06,356 DEBUG [StoreCloser-TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/70cfe190a5154b039755f5fbb2b6ccd5 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/70cfe190a5154b039755f5fbb2b6ccd5 2023-03-29 03:54:06,357 DEBUG [StoreCloser-TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/aa74d84aeca04e5a89a64b97ed2c436a to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/aa74d84aeca04e5a89a64b97ed2c436a 2023-03-29 03:54:06,359 DEBUG [StoreCloser-TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/94d5c73b79ea458c83259402ff270ae6 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/94d5c73b79ea458c83259402ff270ae6 2023-03-29 03:54:06,360 DEBUG [StoreCloser-TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/983f68013b83456da3c3ebebf3213aa9 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/983f68013b83456da3c3ebebf3213aa9 2023-03-29 03:54:06,362 DEBUG [StoreCloser-TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/998dd4287c35476c916f7d249bd5a4fd to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/998dd4287c35476c916f7d249bd5a4fd 2023-03-29 03:54:06,363 DEBUG [StoreCloser-TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/c836889c1b304821a2fdd3333ce7497e to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/c836889c1b304821a2fdd3333ce7497e 2023-03-29 03:54:06,364 DEBUG [StoreCloser-TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/cbd04824c8a54425b408018fb0df02ce to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/cbd04824c8a54425b408018fb0df02ce 2023-03-29 03:54:06,365 DEBUG [StoreCloser-TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/7f5326dd379b4772ab80602b7436ef35 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/7f5326dd379b4772ab80602b7436ef35 2023-03-29 03:54:06,366 DEBUG [StoreCloser-TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/dc998e8829994e0e90aa154e15c64d55 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/dc998e8829994e0e90aa154e15c64d55 2023-03-29 03:54:06,368 DEBUG [StoreCloser-TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/085e330a524048c9bb7cd7e681705b40 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/085e330a524048c9bb7cd7e681705b40 2023-03-29 03:54:06,372 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/recovered.edits/822.seqid, newMaxSeqId=822, maxSeqId=1 2023-03-29 03:54:06,374 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7. 2023-03-29 03:54:06,374 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 52c064cf4695189b45c25208ecad32f7: 2023-03-29 03:54:06,376 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 52c064cf4695189b45c25208ecad32f7 2023-03-29 03:54:06,376 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=52c064cf4695189b45c25208ecad32f7, regionState=CLOSED 2023-03-29 03:54:06,377 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680062046376"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680062046376"}]},"ts":"1680062046376"} 2023-03-29 03:54:06,379 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=14, resume processing ppid=13 2023-03-29 03:54:06,379 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=13, state=SUCCESS; CloseRegionProcedure 52c064cf4695189b45c25208ecad32f7, server=jenkins-hbase4.apache.org,35599,1680062018534 in 1.6140 sec 2023-03-29 03:54:06,381 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=13, resume processing ppid=12 2023-03-29 03:54:06,381 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=52c064cf4695189b45c25208ecad32f7, UNASSIGN in 1.6180 sec 2023-03-29 03:54:06,382 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680062046382"}]},"ts":"1680062046382"} 2023-03-29 03:54:06,383 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-03-29 03:54:06,384 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-03-29 03:54:06,386 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=12, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.6370 sec 2023-03-29 03:54:06,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-03-29 03:54:06,861 INFO [Listener at localhost/35027] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 12 completed 2023-03-29 03:54:06,866 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-03-29 03:54:06,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure2.ProcedureExecutor(1029): Stored pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-03-29 03:54:06,874 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-29 03:54:06,875 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=15, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-29 03:54:06,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-03-29 03:54:06,880 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/.tmp/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7 2023-03-29 03:54:06,882 DEBUG [HFileArchiver-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/.tmp/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A, FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/.tmp/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B, FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/.tmp/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C, FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/.tmp/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/recovered.edits] 2023-03-29 03:54:06,886 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/.tmp/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/5683ee8666c54b20a0c1e8736a95f939 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/5683ee8666c54b20a0c1e8736a95f939 2023-03-29 03:54:06,887 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/.tmp/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/77dff205cf7d4e14a90dd3effe44f57b to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/77dff205cf7d4e14a90dd3effe44f57b 2023-03-29 03:54:06,888 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/.tmp/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/8aafbc195cd54662ad876e05cc2f88e7 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/8aafbc195cd54662ad876e05cc2f88e7 2023-03-29 03:54:06,889 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/.tmp/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/e8f637ff32db41b8b52410273e78f48d to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/A/e8f637ff32db41b8b52410273e78f48d 2023-03-29 03:54:06,892 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/.tmp/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/392347aa60c64425b4e09db17a4a6104 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/392347aa60c64425b4e09db17a4a6104 2023-03-29 03:54:06,893 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/.tmp/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/91df37d95b4d48ed92392ac2fe09d3dc to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/91df37d95b4d48ed92392ac2fe09d3dc 2023-03-29 03:54:06,894 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/.tmp/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/c612972da0584715a4e37d137e048d1d to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/c612972da0584715a4e37d137e048d1d 2023-03-29 03:54:06,896 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/.tmp/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/f87086078e4d42a19bad3b42fffae9c1 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/B/f87086078e4d42a19bad3b42fffae9c1 2023-03-29 03:54:06,898 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/.tmp/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/2c378d2db3934e23b4b17bf4ba6cbb66 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/2c378d2db3934e23b4b17bf4ba6cbb66 2023-03-29 03:54:06,899 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/.tmp/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/67926b042d044834a370b7f68e242a68 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/67926b042d044834a370b7f68e242a68 2023-03-29 03:54:06,901 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/.tmp/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/957cc2ab93574890a90b7f210ce12b6e to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/957cc2ab93574890a90b7f210ce12b6e 2023-03-29 03:54:06,902 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/.tmp/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/dba47320f85f4ef2a160d3e01e8a493c to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/C/dba47320f85f4ef2a160d3e01e8a493c 2023-03-29 03:54:06,904 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/.tmp/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/recovered.edits/822.seqid to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7/recovered.edits/822.seqid 2023-03-29 03:54:06,905 DEBUG [HFileArchiver-3] backup.HFileArchiver(596): Deleted hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/.tmp/data/default/TestAcidGuarantees/52c064cf4695189b45c25208ecad32f7 2023-03-29 03:54:06,905 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-29 03:54:06,907 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=15, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-29 03:54:06,928 WARN [PEWorker-5] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-03-29 03:54:06,973 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-03-29 03:54:06,974 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=15, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-29 03:54:06,974 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-03-29 03:54:06,974 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1680062046974"}]},"ts":"9223372036854775807"} 2023-03-29 03:54:06,977 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-03-29 03:54:06,977 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 52c064cf4695189b45c25208ecad32f7, NAME => 'TestAcidGuarantees,,1680062022177.52c064cf4695189b45c25208ecad32f7.', STARTKEY => '', ENDKEY => ''}] 2023-03-29 03:54:06,977 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-03-29 03:54:06,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-03-29 03:54:06,978 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1680062046977"}]},"ts":"9223372036854775807"} 2023-03-29 03:54:06,980 INFO [PEWorker-5] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-03-29 03:54:06,985 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(130): Finished pid=15, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-29 03:54:06,986 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=15, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 118 msec 2023-03-29 03:54:07,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-03-29 03:54:07,179 INFO [Listener at localhost/35027] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 15 completed 2023-03-29 03:54:07,195 INFO [Listener at localhost/35027] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMixedAtomicity Thread=236 (was 208) Potentially hanging thread: (jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1088250791_15 at /127.0.0.1:51946 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-3 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-7 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS:0;jenkins-hbase4:35599-shortCompactions-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:549) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-13 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1088250791_15 at /127.0.0.1:52004 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-2 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-1088250791_15 at /127.0.0.1:43006 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0xf5645e1-shared-pool-14 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-12 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-10 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0xf5645e1-shared-pool-15 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-1-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-9 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-4 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0xf5645e1-shared-pool-16 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-8 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0xf5645e1-shared-pool-13 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-6 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RSProcedureDispatcher-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-5 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-11 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=576 (was 470) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=528 (was 412) - SystemLoadAverage LEAK? -, ProcessCount=170 (was 170), AvailableMemoryMB=7074 (was 7714) 2023-03-29 03:54:07,206 INFO [Listener at localhost/35027] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobMixedAtomicity Thread=236, OpenFileDescriptor=576, MaxFileDescriptor=60000, SystemLoadAverage=528, ProcessCount=170, AvailableMemoryMB=7074 2023-03-29 03:54:07,208 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-29 03:54:07,209 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-29 03:54:07,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure2.ProcedureExecutor(1029): Stored pid=16, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-03-29 03:54:07,211 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-03-29 03:54:07,211 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 16 2023-03-29 03:54:07,212 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-03-29 03:54:07,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-03-29 03:54:07,213 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/.tmp/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0 2023-03-29 03:54:07,214 DEBUG [HFileArchiver-4] backup.HFileArchiver(153): Directory hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/.tmp/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0 empty. 2023-03-29 03:54:07,214 DEBUG [HFileArchiver-4] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/.tmp/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0 2023-03-29 03:54:07,214 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-29 03:54:07,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-03-29 03:54:07,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-03-29 03:54:07,622 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-29 03:54:07,624 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => ee0c8ff38ef29b37b45871dfc89897f0, NAME => 'TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/.tmp 2023-03-29 03:54:07,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-03-29 03:54:08,031 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-29 03:54:08,031 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing ee0c8ff38ef29b37b45871dfc89897f0, disabling compactions & flushes 2023-03-29 03:54:08,031 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. 2023-03-29 03:54:08,031 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. 2023-03-29 03:54:08,031 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. after waiting 0 ms 2023-03-29 03:54:08,031 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. 2023-03-29 03:54:08,031 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. 2023-03-29 03:54:08,031 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for ee0c8ff38ef29b37b45871dfc89897f0: 2023-03-29 03:54:08,033 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-03-29 03:54:08,034 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680062048034"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680062048034"}]},"ts":"1680062048034"} 2023-03-29 03:54:08,036 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-03-29 03:54:08,037 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-03-29 03:54:08,037 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680062048037"}]},"ts":"1680062048037"} 2023-03-29 03:54:08,038 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-03-29 03:54:08,041 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=ee0c8ff38ef29b37b45871dfc89897f0, ASSIGN}] 2023-03-29 03:54:08,043 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=ee0c8ff38ef29b37b45871dfc89897f0, ASSIGN 2023-03-29 03:54:08,044 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=ee0c8ff38ef29b37b45871dfc89897f0, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,35599,1680062018534; forceNewPlan=false, retain=false 2023-03-29 03:54:08,195 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=ee0c8ff38ef29b37b45871dfc89897f0, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:08,195 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680062048195"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680062048195"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680062048195"}]},"ts":"1680062048195"} 2023-03-29 03:54:08,197 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=18, ppid=17, state=RUNNABLE; OpenRegionProcedure ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534}] 2023-03-29 03:54:08,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-03-29 03:54:08,353 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. 2023-03-29 03:54:08,353 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => ee0c8ff38ef29b37b45871dfc89897f0, NAME => 'TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0.', STARTKEY => '', ENDKEY => ''} 2023-03-29 03:54:08,353 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees ee0c8ff38ef29b37b45871dfc89897f0 2023-03-29 03:54:08,353 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-29 03:54:08,353 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for ee0c8ff38ef29b37b45871dfc89897f0 2023-03-29 03:54:08,354 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for ee0c8ff38ef29b37b45871dfc89897f0 2023-03-29 03:54:08,355 INFO [StoreOpener-ee0c8ff38ef29b37b45871dfc89897f0-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region ee0c8ff38ef29b37b45871dfc89897f0 2023-03-29 03:54:08,356 DEBUG [StoreOpener-ee0c8ff38ef29b37b45871dfc89897f0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A 2023-03-29 03:54:08,356 DEBUG [StoreOpener-ee0c8ff38ef29b37b45871dfc89897f0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A 2023-03-29 03:54:08,356 INFO [StoreOpener-ee0c8ff38ef29b37b45871dfc89897f0-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-29 03:54:08,356 INFO [StoreOpener-ee0c8ff38ef29b37b45871dfc89897f0-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ee0c8ff38ef29b37b45871dfc89897f0 columnFamilyName A 2023-03-29 03:54:08,357 INFO [StoreOpener-ee0c8ff38ef29b37b45871dfc89897f0-1] regionserver.HStore(310): Store=ee0c8ff38ef29b37b45871dfc89897f0/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-29 03:54:08,357 INFO [StoreOpener-ee0c8ff38ef29b37b45871dfc89897f0-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region ee0c8ff38ef29b37b45871dfc89897f0 2023-03-29 03:54:08,358 DEBUG [StoreOpener-ee0c8ff38ef29b37b45871dfc89897f0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B 2023-03-29 03:54:08,358 DEBUG [StoreOpener-ee0c8ff38ef29b37b45871dfc89897f0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B 2023-03-29 03:54:08,358 INFO [StoreOpener-ee0c8ff38ef29b37b45871dfc89897f0-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-29 03:54:08,359 INFO [StoreOpener-ee0c8ff38ef29b37b45871dfc89897f0-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ee0c8ff38ef29b37b45871dfc89897f0 columnFamilyName B 2023-03-29 03:54:08,359 INFO [StoreOpener-ee0c8ff38ef29b37b45871dfc89897f0-1] regionserver.HStore(310): Store=ee0c8ff38ef29b37b45871dfc89897f0/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-29 03:54:08,359 INFO [StoreOpener-ee0c8ff38ef29b37b45871dfc89897f0-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region ee0c8ff38ef29b37b45871dfc89897f0 2023-03-29 03:54:08,361 DEBUG [StoreOpener-ee0c8ff38ef29b37b45871dfc89897f0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C 2023-03-29 03:54:08,361 DEBUG [StoreOpener-ee0c8ff38ef29b37b45871dfc89897f0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C 2023-03-29 03:54:08,361 INFO [StoreOpener-ee0c8ff38ef29b37b45871dfc89897f0-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-29 03:54:08,361 INFO [StoreOpener-ee0c8ff38ef29b37b45871dfc89897f0-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ee0c8ff38ef29b37b45871dfc89897f0 columnFamilyName C 2023-03-29 03:54:08,361 INFO [StoreOpener-ee0c8ff38ef29b37b45871dfc89897f0-1] regionserver.HStore(310): Store=ee0c8ff38ef29b37b45871dfc89897f0/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-29 03:54:08,361 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. 2023-03-29 03:54:08,362 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0 2023-03-29 03:54:08,363 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0 2023-03-29 03:54:08,365 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-29 03:54:08,366 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for ee0c8ff38ef29b37b45871dfc89897f0 2023-03-29 03:54:08,368 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-29 03:54:08,369 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened ee0c8ff38ef29b37b45871dfc89897f0; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61246377, jitterRate=-0.08735786378383636}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-29 03:54:08,369 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for ee0c8ff38ef29b37b45871dfc89897f0: 2023-03-29 03:54:08,369 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0., pid=18, masterSystemTime=1680062048349 2023-03-29 03:54:08,371 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. 2023-03-29 03:54:08,371 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. 2023-03-29 03:54:08,371 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=ee0c8ff38ef29b37b45871dfc89897f0, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:08,372 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680062048371"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1680062048371"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680062048371"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680062048371"}]},"ts":"1680062048371"} 2023-03-29 03:54:08,374 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=18, resume processing ppid=17 2023-03-29 03:54:08,374 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=18, ppid=17, state=SUCCESS; OpenRegionProcedure ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 in 176 msec 2023-03-29 03:54:08,376 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=17, resume processing ppid=16 2023-03-29 03:54:08,376 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=17, ppid=16, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=ee0c8ff38ef29b37b45871dfc89897f0, ASSIGN in 333 msec 2023-03-29 03:54:08,376 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-03-29 03:54:08,376 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680062048376"}]},"ts":"1680062048376"} 2023-03-29 03:54:08,377 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-03-29 03:54:08,381 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-03-29 03:54:08,382 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=16, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1720 sec 2023-03-29 03:54:09,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-03-29 03:54:09,318 INFO [Listener at localhost/35027] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 16 completed 2023-03-29 03:54:09,319 DEBUG [Listener at localhost/35027] zookeeper.ReadOnlyZKClient(139): Connect 0x324dc814 to 127.0.0.1:52103 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-29 03:54:09,324 DEBUG [Listener at localhost/35027] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5a301dcb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-29 03:54:09,326 DEBUG [Listener at localhost/35027] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-29 03:54:09,330 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40102, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-29 03:54:09,332 DEBUG [Listener at localhost/35027] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-03-29 03:54:09,337 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52718, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-03-29 03:54:09,344 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-29 03:54:09,345 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.HMaster$12(2454): Client=jenkins//172.31.14.131 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-29 03:54:09,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure2.ProcedureExecutor(1029): Stored pid=19, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-03-29 03:54:09,776 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-03-29 03:54:09,777 DEBUG [PEWorker-4] util.FSTableDescriptors(519): Deleted hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-29 03:54:09,777 INFO [PEWorker-4] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-03-29 03:54:09,781 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=20, ppid=19, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-03-29 03:54:09,793 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=ee0c8ff38ef29b37b45871dfc89897f0, REOPEN/MOVE}] 2023-03-29 03:54:09,794 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=ee0c8ff38ef29b37b45871dfc89897f0, REOPEN/MOVE 2023-03-29 03:54:09,794 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=ee0c8ff38ef29b37b45871dfc89897f0, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:09,794 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680062049794"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680062049794"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680062049794"}]},"ts":"1680062049794"} 2023-03-29 03:54:09,796 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=22, ppid=21, state=RUNNABLE; CloseRegionProcedure ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534}] 2023-03-29 03:54:09,948 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close ee0c8ff38ef29b37b45871dfc89897f0 2023-03-29 03:54:09,949 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing ee0c8ff38ef29b37b45871dfc89897f0, disabling compactions & flushes 2023-03-29 03:54:09,949 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. 2023-03-29 03:54:09,949 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. 2023-03-29 03:54:09,949 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. after waiting 0 ms 2023-03-29 03:54:09,949 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. 2023-03-29 03:54:09,952 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-03-29 03:54:09,953 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. 2023-03-29 03:54:09,953 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for ee0c8ff38ef29b37b45871dfc89897f0: 2023-03-29 03:54:09,953 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: ee0c8ff38ef29b37b45871dfc89897f0 to self. 2023-03-29 03:54:09,954 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed ee0c8ff38ef29b37b45871dfc89897f0 2023-03-29 03:54:09,955 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=ee0c8ff38ef29b37b45871dfc89897f0, regionState=CLOSED 2023-03-29 03:54:09,955 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680062049955"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680062049955"}]},"ts":"1680062049955"} 2023-03-29 03:54:09,958 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=22, resume processing ppid=21 2023-03-29 03:54:09,958 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=22, ppid=21, state=SUCCESS; CloseRegionProcedure ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 in 160 msec 2023-03-29 03:54:09,959 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=ee0c8ff38ef29b37b45871dfc89897f0, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,35599,1680062018534; forceNewPlan=false, retain=true 2023-03-29 03:54:10,109 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=ee0c8ff38ef29b37b45871dfc89897f0, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:10,109 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680062050109"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680062050109"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680062050109"}]},"ts":"1680062050109"} 2023-03-29 03:54:10,111 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=23, ppid=21, state=RUNNABLE; OpenRegionProcedure ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534}] 2023-03-29 03:54:10,266 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. 2023-03-29 03:54:10,266 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => ee0c8ff38ef29b37b45871dfc89897f0, NAME => 'TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0.', STARTKEY => '', ENDKEY => ''} 2023-03-29 03:54:10,267 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees ee0c8ff38ef29b37b45871dfc89897f0 2023-03-29 03:54:10,267 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-29 03:54:10,267 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for ee0c8ff38ef29b37b45871dfc89897f0 2023-03-29 03:54:10,267 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for ee0c8ff38ef29b37b45871dfc89897f0 2023-03-29 03:54:10,269 INFO [StoreOpener-ee0c8ff38ef29b37b45871dfc89897f0-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region ee0c8ff38ef29b37b45871dfc89897f0 2023-03-29 03:54:10,270 DEBUG [StoreOpener-ee0c8ff38ef29b37b45871dfc89897f0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A 2023-03-29 03:54:10,270 DEBUG [StoreOpener-ee0c8ff38ef29b37b45871dfc89897f0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A 2023-03-29 03:54:10,270 INFO [StoreOpener-ee0c8ff38ef29b37b45871dfc89897f0-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-29 03:54:10,273 INFO [StoreOpener-ee0c8ff38ef29b37b45871dfc89897f0-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ee0c8ff38ef29b37b45871dfc89897f0 columnFamilyName A 2023-03-29 03:54:10,275 INFO [StoreOpener-ee0c8ff38ef29b37b45871dfc89897f0-1] regionserver.HStore(310): Store=ee0c8ff38ef29b37b45871dfc89897f0/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-29 03:54:10,275 INFO [StoreOpener-ee0c8ff38ef29b37b45871dfc89897f0-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region ee0c8ff38ef29b37b45871dfc89897f0 2023-03-29 03:54:10,276 DEBUG [StoreOpener-ee0c8ff38ef29b37b45871dfc89897f0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B 2023-03-29 03:54:10,276 DEBUG [StoreOpener-ee0c8ff38ef29b37b45871dfc89897f0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B 2023-03-29 03:54:10,277 INFO [StoreOpener-ee0c8ff38ef29b37b45871dfc89897f0-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-29 03:54:10,277 INFO [StoreOpener-ee0c8ff38ef29b37b45871dfc89897f0-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ee0c8ff38ef29b37b45871dfc89897f0 columnFamilyName B 2023-03-29 03:54:10,277 INFO [StoreOpener-ee0c8ff38ef29b37b45871dfc89897f0-1] regionserver.HStore(310): Store=ee0c8ff38ef29b37b45871dfc89897f0/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-29 03:54:10,278 INFO [StoreOpener-ee0c8ff38ef29b37b45871dfc89897f0-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region ee0c8ff38ef29b37b45871dfc89897f0 2023-03-29 03:54:10,278 DEBUG [StoreOpener-ee0c8ff38ef29b37b45871dfc89897f0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C 2023-03-29 03:54:10,278 DEBUG [StoreOpener-ee0c8ff38ef29b37b45871dfc89897f0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C 2023-03-29 03:54:10,278 INFO [StoreOpener-ee0c8ff38ef29b37b45871dfc89897f0-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-29 03:54:10,279 INFO [StoreOpener-ee0c8ff38ef29b37b45871dfc89897f0-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region ee0c8ff38ef29b37b45871dfc89897f0 columnFamilyName C 2023-03-29 03:54:10,280 INFO [StoreOpener-ee0c8ff38ef29b37b45871dfc89897f0-1] regionserver.HStore(310): Store=ee0c8ff38ef29b37b45871dfc89897f0/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-29 03:54:10,280 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. 2023-03-29 03:54:10,280 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0 2023-03-29 03:54:10,281 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0 2023-03-29 03:54:10,283 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-29 03:54:10,284 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for ee0c8ff38ef29b37b45871dfc89897f0 2023-03-29 03:54:10,285 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened ee0c8ff38ef29b37b45871dfc89897f0; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=75323132, jitterRate=0.1224021315574646}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-29 03:54:10,285 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for ee0c8ff38ef29b37b45871dfc89897f0: 2023-03-29 03:54:10,285 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0., pid=23, masterSystemTime=1680062050263 2023-03-29 03:54:10,287 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. 2023-03-29 03:54:10,287 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. 2023-03-29 03:54:10,287 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=ee0c8ff38ef29b37b45871dfc89897f0, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:10,288 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680062050287"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1680062050287"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680062050287"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680062050287"}]},"ts":"1680062050287"} 2023-03-29 03:54:10,290 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=23, resume processing ppid=21 2023-03-29 03:54:10,290 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=23, ppid=21, state=SUCCESS; OpenRegionProcedure ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 in 178 msec 2023-03-29 03:54:10,291 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=21, resume processing ppid=20 2023-03-29 03:54:10,291 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=21, ppid=20, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=ee0c8ff38ef29b37b45871dfc89897f0, REOPEN/MOVE in 497 msec 2023-03-29 03:54:10,294 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=20, resume processing ppid=19 2023-03-29 03:54:10,294 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=20, ppid=19, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 512 msec 2023-03-29 03:54:10,296 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=19, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 945 msec 2023-03-29 03:54:10,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=19 2023-03-29 03:54:10,304 DEBUG [Listener at localhost/35027] zookeeper.ReadOnlyZKClient(139): Connect 0x55cbb115 to 127.0.0.1:52103 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-29 03:54:10,310 DEBUG [Listener at localhost/35027] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1aa5eef5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-29 03:54:10,311 DEBUG [Listener at localhost/35027] zookeeper.ReadOnlyZKClient(139): Connect 0x24bd3872 to 127.0.0.1:52103 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-29 03:54:10,315 DEBUG [Listener at localhost/35027] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@73e51571, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-29 03:54:10,315 DEBUG [Listener at localhost/35027] zookeeper.ReadOnlyZKClient(139): Connect 0x7aba8e46 to 127.0.0.1:52103 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-29 03:54:10,319 DEBUG [Listener at localhost/35027] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@370947cb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-29 03:54:10,319 DEBUG [Listener at localhost/35027] zookeeper.ReadOnlyZKClient(139): Connect 0x210d1008 to 127.0.0.1:52103 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-29 03:54:10,324 DEBUG [Listener at localhost/35027] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5c472cfd, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-29 03:54:10,324 DEBUG [Listener at localhost/35027] zookeeper.ReadOnlyZKClient(139): Connect 0x00c2598b to 127.0.0.1:52103 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-29 03:54:10,329 DEBUG [Listener at localhost/35027] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@498694cb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-29 03:54:10,329 DEBUG [Listener at localhost/35027] zookeeper.ReadOnlyZKClient(139): Connect 0x421e5356 to 127.0.0.1:52103 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-29 03:54:10,333 DEBUG [Listener at localhost/35027] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3f43b712, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-29 03:54:10,333 DEBUG [Listener at localhost/35027] zookeeper.ReadOnlyZKClient(139): Connect 0x4a0fd7b4 to 127.0.0.1:52103 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-29 03:54:10,336 DEBUG [Listener at localhost/35027] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7a3bb940, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-29 03:54:10,337 DEBUG [Listener at localhost/35027] zookeeper.ReadOnlyZKClient(139): Connect 0x1a7b66a3 to 127.0.0.1:52103 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-29 03:54:10,340 DEBUG [Listener at localhost/35027] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2efdb02f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-29 03:54:10,340 DEBUG [Listener at localhost/35027] zookeeper.ReadOnlyZKClient(139): Connect 0x42d325f4 to 127.0.0.1:52103 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-29 03:54:10,345 DEBUG [Listener at localhost/35027] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3dc1d73, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-29 03:54:10,350 DEBUG [hconnection-0x50407ad2-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-29 03:54:10,350 DEBUG [hconnection-0x703cdf85-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-29 03:54:10,351 DEBUG [hconnection-0x4f0ccb14-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-29 03:54:10,351 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:54:10,352 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40112, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-29 03:54:10,353 DEBUG [hconnection-0x45f6828f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-29 03:54:10,357 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40120, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-29 03:54:10,357 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40124, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-29 03:54:10,357 DEBUG [hconnection-0x5d4811ff-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-29 03:54:10,357 DEBUG [hconnection-0x3062b7ab-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-29 03:54:10,357 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40130, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-29 03:54:10,360 DEBUG [hconnection-0x2539490-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-29 03:54:10,362 DEBUG [hconnection-0x2c12cde8-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-29 03:54:10,362 DEBUG [hconnection-0xeccda98-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-29 03:54:10,363 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40142, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-29 03:54:10,363 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40148, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-29 03:54:10,365 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40152, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-29 03:54:10,365 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40144, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-29 03:54:10,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:54:10,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:54:10,378 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:54:10,378 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:10,378 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:54:10,378 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:54:10,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on ee0c8ff38ef29b37b45871dfc89897f0 2023-03-29 03:54:10,380 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:10,380 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:10,380 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ee0c8ff38ef29b37b45871dfc89897f0 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-29 03:54:10,380 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee0c8ff38ef29b37b45871dfc89897f0, store=A 2023-03-29 03:54:10,380 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:54:10,380 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee0c8ff38ef29b37b45871dfc89897f0, store=B 2023-03-29 03:54:10,380 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:54:10,380 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee0c8ff38ef29b37b45871dfc89897f0, store=C 2023-03-29 03:54:10,380 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:54:10,381 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:10,381 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:10,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:10,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:10,382 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:10,382 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:54:10,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:10,382 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:10,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:54:10,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:10,382 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:54:10,388 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:54:10,389 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:54:10,389 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:54:10,389 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:10,389 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:54:10,390 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40154, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-29 03:54:10,390 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:54:10,390 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:54:10,395 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. 2023-03-29 03:54:10,395 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. started... 2023-03-29 03:54:10,395 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. as already flushing 2023-03-29 03:54:10,396 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. 2023-03-29 03:54:10,396 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:54:10,396 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:54:10,396 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:10,396 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:54:10,396 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:54:10,400 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:10,400 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:10,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:10,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:10,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:10,401 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:10,401 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:54:10,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:10,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:10,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:10,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:10,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:10,409 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:54:10,409 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b29784c[Count = 0] remaining members to acquire global barrier 2023-03-29 03:54:10,409 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:54:10,409 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:10,410 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:10,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:10,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:10,410 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:54:10,410 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:54:10,410 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:54:10,411 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:10,411 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:54:10,413 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:54:10,413 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:10,413 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:10,413 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:54:10,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:10,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:10,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:10,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:10,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:10,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:10,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:10,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:10,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:10,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:10,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:54:10,417 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:54:10,417 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:54:10,417 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:54:10,417 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:54:10,417 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:54:10,417 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:54:10,419 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:10,419 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:10,419 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:10,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:10,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:10,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:10,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:10,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:10,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:10,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:10,423 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:10,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:10,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:10,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:10,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:10,425 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:10,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:10,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:10,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:10,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:10,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:10,428 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:10,428 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:10,428 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:10,428 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:54:10,429 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:10,429 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:54:10,429 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:54:10,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:10,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:10,428 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:10,429 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:10,429 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:10,429 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:10,429 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:10,429 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:10,429 DEBUG [Thread-985] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:54:10,429 DEBUG [Thread-985] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:54:10,429 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:54:10,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:10,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:10,443 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680062050415, max timestamp=1680062050437Num uniques 859; 2023-03-29 03:54:10,443 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680062050415, max timestamp=1680062050437Num uniques 859; 2023-03-29 03:54:10,443 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-29 03:54:10,443 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-29 03:54:10,445 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680062050415, max timestamp=1680062050437Num uniques 859; 2023-03-29 03:54:10,445 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-29 03:54:10,490 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:10,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40120 deadline: 1680062110474, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:10,491 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:10,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40148 deadline: 1680062110487, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:10,495 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303292ac6c3e5a4e34171b89245b9d885f590 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303292ac6c3e5a4e34171b89245b9d885f590 2023-03-29 03:54:10,502 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=16, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/A/a8fc006fc607439e884193ed31339f0d 2023-03-29 03:54:10,506 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:10,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40124 deadline: 1680062110491, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:10,507 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:10,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40142 deadline: 1680062110491, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:10,508 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:10,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40112 deadline: 1680062110492, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:10,529 DEBUG [Thread-985] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:54:10,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:54:10,535 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:54:10,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:54:10,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:54:10,538 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:54:10,538 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:10,538 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:54:10,538 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:54:10,538 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:10,538 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:10,541 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:10,541 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:10,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:10,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:10,541 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:10,541 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:54:10,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:10,542 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:10,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:54:10,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:10,542 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:54:10,543 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:54:10,543 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:54:10,543 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:54:10,543 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:10,543 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:54:10,544 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:54:10,544 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:54:10,544 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. 2023-03-29 03:54:10,544 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. started... 2023-03-29 03:54:10,544 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. as already flushing 2023-03-29 03:54:10,544 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. 2023-03-29 03:54:10,544 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:54:10,544 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:54:10,544 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:10,544 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:54:10,544 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:54:10,549 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:10,549 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:10,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:10,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:10,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:10,550 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:10,550 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:54:10,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:10,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:10,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:10,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:10,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:10,551 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:54:10,552 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@795e7944[Count = 0] remaining members to acquire global barrier 2023-03-29 03:54:10,552 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:54:10,552 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:10,553 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:10,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:10,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:10,553 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:54:10,553 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:54:10,553 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:54:10,553 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:10,554 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:54:10,555 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:54:10,555 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:10,555 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:10,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:10,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:10,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:10,555 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:54:10,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:10,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:10,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:10,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:10,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:10,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:10,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:10,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:54:10,558 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:54:10,558 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:54:10,559 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:54:10,559 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:54:10,559 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:54:10,559 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:54:10,564 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:10,564 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:10,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:10,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:10,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:10,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:10,564 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:10,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:10,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:10,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:10,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:10,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:10,567 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:10,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:10,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:10,570 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:10,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:10,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:10,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:10,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:10,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:10,574 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:10,574 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:10,574 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:10,574 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:10,574 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:10,574 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:10,574 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:54:10,574 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:54:10,574 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:10,575 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:10,575 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:10,575 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:10,575 DEBUG [Thread-985] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:54:10,575 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:54:10,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:10,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:10,575 DEBUG [Thread-985] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:54:10,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:10,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:10,597 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:10,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40120 deadline: 1680062110593, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:10,597 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:10,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40148 deadline: 1680062110596, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:10,610 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:10,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40124 deadline: 1680062110609, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:10,610 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:10,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40142 deadline: 1680062110609, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:10,611 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:10,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40112 deadline: 1680062110609, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:10,675 DEBUG [Thread-985] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:54:10,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:54:10,681 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:54:10,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:54:10,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:54:10,692 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:54:10,692 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:10,692 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:54:10,692 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:54:10,693 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:10,693 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:10,696 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:10,696 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:10,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:10,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:10,696 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:10,696 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:54:10,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:10,697 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:10,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:54:10,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:10,697 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:54:10,700 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:54:10,700 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:54:10,700 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:54:10,700 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:10,700 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:54:10,700 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:54:10,700 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:54:10,700 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. 2023-03-29 03:54:10,700 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. started... 2023-03-29 03:54:10,700 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. as already flushing 2023-03-29 03:54:10,700 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. 2023-03-29 03:54:10,701 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:54:10,701 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:54:10,701 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:10,701 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:54:10,701 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:54:10,702 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:10,702 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:10,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:10,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:10,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:10,703 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:10,703 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:54:10,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:10,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:10,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:10,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:10,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:10,704 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:54:10,704 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a4b199d[Count = 0] remaining members to acquire global barrier 2023-03-29 03:54:10,704 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:54:10,704 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:10,706 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:10,706 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:54:10,707 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:10,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:10,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:10,707 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:54:10,707 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:54:10,707 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:54:10,709 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:54:10,709 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:10,709 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:10,709 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:54:10,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:10,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:10,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:10,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:10,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:10,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:10,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:10,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:10,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:10,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:10,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:54:10,712 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:54:10,712 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:54:10,712 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:54:10,712 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:54:10,712 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:54:10,712 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:54:10,718 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:10,718 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:10,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:10,718 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:10,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:10,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:10,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:10,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:10,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:10,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:10,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:10,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:10,719 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:10,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:10,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:10,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:10,723 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:10,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:10,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:10,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:10,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:10,727 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:10,727 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:10,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:10,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:10,727 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:10,727 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:10,727 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:10,727 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:10,727 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:10,727 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:10,727 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:10,727 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:10,727 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:54:10,728 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:54:10,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:10,728 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:54:10,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:10,728 DEBUG [Thread-985] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:54:10,728 DEBUG [Thread-985] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:54:10,803 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:10,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40148 deadline: 1680062110799, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:10,804 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:10,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40120 deadline: 1680062110800, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:10,813 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:10,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40142 deadline: 1680062110813, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:10,819 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:10,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40124 deadline: 1680062110813, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:10,819 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:10,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40112 deadline: 1680062110813, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:10,828 DEBUG [Thread-985] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:54:10,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:54:10,831 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:54:10,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:54:10,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:54:10,833 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:54:10,833 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:10,833 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:54:10,833 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:54:10,834 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:10,834 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:10,836 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:10,836 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:10,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:10,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:10,836 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:10,836 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:54:10,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:10,837 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:10,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:54:10,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:10,837 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:54:10,838 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:54:10,838 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:54:10,838 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:54:10,838 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:10,838 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:54:10,840 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:54:10,840 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:54:10,840 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. 2023-03-29 03:54:10,840 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. started... 2023-03-29 03:54:10,840 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. as already flushing 2023-03-29 03:54:10,840 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. 2023-03-29 03:54:10,840 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:54:10,840 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:54:10,840 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:10,840 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:54:10,840 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:54:10,843 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:10,843 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:10,843 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:54:10,848 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:10,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:10,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:10,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:10,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:10,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:10,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:10,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:10,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:10,850 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:54:10,850 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3b99118f[Count = 0] remaining members to acquire global barrier 2023-03-29 03:54:10,850 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:54:10,850 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:10,853 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:10,853 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:54:10,853 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:10,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:10,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:10,854 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:54:10,854 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:54:10,854 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:54:10,855 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:10,855 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:54:10,855 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:10,855 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:54:10,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:10,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:10,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:10,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:10,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:10,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:10,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:10,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:10,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:10,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:10,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:54:10,858 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:54:10,858 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:54:10,858 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:54:10,858 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:54:10,858 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:54:10,858 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:54:10,860 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:10,860 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:10,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:10,860 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:10,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:10,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:10,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:10,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:10,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:10,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:10,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:10,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:10,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:10,862 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:10,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:10,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:10,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:10,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:10,864 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:10,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:10,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:10,866 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:10,867 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:10,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:10,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:10,867 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:10,867 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:10,867 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:10,867 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:54:10,867 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:54:10,867 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:10,867 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:10,867 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:10,867 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:10,867 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:10,867 DEBUG [Thread-985] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:54:10,867 DEBUG [Thread-985] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:54:10,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:10,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:10,927 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=16 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/B/2c0c3cd702474babb481d56d8248c279 2023-03-29 03:54:10,950 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=16 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/C/db71cfdcf2df4477b59fafaae3b40be2 2023-03-29 03:54:10,955 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/A/a8fc006fc607439e884193ed31339f0d as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/a8fc006fc607439e884193ed31339f0d 2023-03-29 03:54:10,961 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/a8fc006fc607439e884193ed31339f0d, entries=150, sequenceid=16, filesize=25.1 K 2023-03-29 03:54:10,964 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/B/2c0c3cd702474babb481d56d8248c279 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/2c0c3cd702474babb481d56d8248c279 2023-03-29 03:54:10,967 DEBUG [Thread-985] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:54:10,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:54:10,971 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:54:10,978 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/2c0c3cd702474babb481d56d8248c279, entries=150, sequenceid=16, filesize=11.6 K 2023-03-29 03:54:10,979 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/C/db71cfdcf2df4477b59fafaae3b40be2 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/db71cfdcf2df4477b59fafaae3b40be2 2023-03-29 03:54:10,983 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/db71cfdcf2df4477b59fafaae3b40be2, entries=150, sequenceid=16, filesize=11.6 K 2023-03-29 03:54:10,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:54:10,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:54:10,984 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:54:10,984 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:10,984 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:54:10,984 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:54:10,984 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=134.18 KB/137400 for ee0c8ff38ef29b37b45871dfc89897f0 in 604ms, sequenceid=16, compaction requested=false 2023-03-29 03:54:10,984 DEBUG [MemStoreFlusher.0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-03-29 03:54:10,984 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:10,984 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:10,985 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ee0c8ff38ef29b37b45871dfc89897f0: 2023-03-29 03:54:10,987 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:10,987 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:10,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:10,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:10,988 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:10,988 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:54:10,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:10,988 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:10,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:54:10,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:10,989 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:54:10,992 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:54:10,992 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:54:10,992 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:54:10,992 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:10,992 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:54:10,992 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:54:10,992 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:54:10,992 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. 2023-03-29 03:54:10,992 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. started... 2023-03-29 03:54:10,992 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ee0c8ff38ef29b37b45871dfc89897f0 3/3 column families, dataSize=134.18 KB heapSize=375.61 KB 2023-03-29 03:54:10,993 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee0c8ff38ef29b37b45871dfc89897f0, store=A 2023-03-29 03:54:10,993 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-29 03:54:10,993 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee0c8ff38ef29b37b45871dfc89897f0, store=B 2023-03-29 03:54:10,993 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-29 03:54:10,993 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee0c8ff38ef29b37b45871dfc89897f0, store=C 2023-03-29 03:54:10,993 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-29 03:54:11,034 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230329cd337a000bbe48d4a04704fc0cae1167 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230329cd337a000bbe48d4a04704fc0cae1167 2023-03-29 03:54:11,047 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=51, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/A/310be1c32da64f1295d33cad9bb2c787 2023-03-29 03:54:11,100 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=51 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/B/1b91a4cdcb154fb3ad48aa9f4dcd49c0 2023-03-29 03:54:11,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on ee0c8ff38ef29b37b45871dfc89897f0 2023-03-29 03:54:11,111 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. as already flushing 2023-03-29 03:54:11,119 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:11,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40148 deadline: 1680062111118, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:11,120 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:11,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40120 deadline: 1680062111120, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:11,120 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:11,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40142 deadline: 1680062111120, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:11,121 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:11,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40124 deadline: 1680062111120, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:11,123 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:11,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40112 deadline: 1680062111123, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:11,131 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=51 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/C/c13f379d2c624f74bce0bc02627e55e0 2023-03-29 03:54:11,138 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/A/310be1c32da64f1295d33cad9bb2c787 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/310be1c32da64f1295d33cad9bb2c787 2023-03-29 03:54:11,143 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/310be1c32da64f1295d33cad9bb2c787, entries=150, sequenceid=51, filesize=25.1 K 2023-03-29 03:54:11,144 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/B/1b91a4cdcb154fb3ad48aa9f4dcd49c0 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/1b91a4cdcb154fb3ad48aa9f4dcd49c0 2023-03-29 03:54:11,154 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/1b91a4cdcb154fb3ad48aa9f4dcd49c0, entries=150, sequenceid=51, filesize=11.6 K 2023-03-29 03:54:11,156 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/C/c13f379d2c624f74bce0bc02627e55e0 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/c13f379d2c624f74bce0bc02627e55e0 2023-03-29 03:54:11,164 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/c13f379d2c624f74bce0bc02627e55e0, entries=150, sequenceid=51, filesize=11.6 K 2023-03-29 03:54:11,165 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~375.56 KB/384576, currentSize=46.96 KB/48090 for ee0c8ff38ef29b37b45871dfc89897f0 in 173ms, sequenceid=51, compaction requested=false 2023-03-29 03:54:11,169 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ee0c8ff38ef29b37b45871dfc89897f0: 2023-03-29 03:54:11,169 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. 2023-03-29 03:54:11,169 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:54:11,169 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:54:11,169 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:11,169 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:54:11,169 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:54:11,171 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:11,171 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:11,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:11,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:11,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:11,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:11,172 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:11,172 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:54:11,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:11,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:11,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:11,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:11,175 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:54:11,175 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ba2e5a8[Count = 0] remaining members to acquire global barrier 2023-03-29 03:54:11,175 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:54:11,175 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:11,176 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:11,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:11,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:11,177 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:54:11,177 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:54:11,177 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:54:11,177 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:11,177 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:54:11,179 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:54:11,179 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:11,179 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:54:11,180 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:11,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:11,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:11,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:11,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:11,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:11,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:11,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:11,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:11,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:11,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:11,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:54:11,183 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:54:11,183 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:54:11,183 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:54:11,183 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:54:11,183 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:54:11,183 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:54:11,185 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:11,185 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:11,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:11,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:11,185 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:11,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:11,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:11,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:11,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:11,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:11,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:11,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:11,187 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:11,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:11,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:11,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:11,188 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:11,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:11,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:11,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:11,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:11,198 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:11,198 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:11,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:11,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:11,198 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:11,198 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:11,198 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:11,198 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:11,198 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:11,198 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:11,198 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:11,198 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:54:11,198 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:54:11,198 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:11,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:11,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:11,199 DEBUG [Thread-985] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:54:11,199 DEBUG [Thread-985] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:54:11,223 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ee0c8ff38ef29b37b45871dfc89897f0 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-29 03:54:11,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on ee0c8ff38ef29b37b45871dfc89897f0 2023-03-29 03:54:11,225 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee0c8ff38ef29b37b45871dfc89897f0, store=A 2023-03-29 03:54:11,225 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:54:11,225 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee0c8ff38ef29b37b45871dfc89897f0, store=B 2023-03-29 03:54:11,225 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:54:11,225 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee0c8ff38ef29b37b45871dfc89897f0, store=C 2023-03-29 03:54:11,225 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:54:11,258 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:11,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40148 deadline: 1680062111254, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:11,259 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:11,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40142 deadline: 1680062111258, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:11,259 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:11,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40120 deadline: 1680062111259, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:11,260 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680062051243, max timestamp=1680062051257Num uniques 859; 2023-03-29 03:54:11,260 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-29 03:54:11,262 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680062051243, max timestamp=1680062051257Num uniques 859; 2023-03-29 03:54:11,262 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-29 03:54:11,263 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680062051243, max timestamp=1680062051257Num uniques 859; 2023-03-29 03:54:11,263 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-29 03:54:11,270 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032933047437159840588e1b67a8cca0d4f4 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032933047437159840588e1b67a8cca0d4f4 2023-03-29 03:54:11,279 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=63, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/A/9c01fbdb1bda447ab51e87c017b39dd9 2023-03-29 03:54:11,299 DEBUG [Thread-985] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:54:11,299 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=63 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/B/7650c3e49e204614842a42aaff888b4c 2023-03-29 03:54:11,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:54:11,302 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:54:11,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:54:11,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:54:11,324 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:54:11,324 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:11,324 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:54:11,324 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:54:11,326 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:11,326 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:11,328 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:11,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:11,328 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:11,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:11,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:11,328 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:11,328 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:54:11,329 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:11,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:54:11,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:11,329 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:54:11,337 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:54:11,337 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:54:11,337 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:54:11,337 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:11,337 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:54:11,342 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:54:11,342 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:54:11,342 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. 2023-03-29 03:54:11,342 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. started... 2023-03-29 03:54:11,342 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. as already flushing 2023-03-29 03:54:11,342 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. 2023-03-29 03:54:11,343 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:54:11,343 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:54:11,343 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:11,343 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:54:11,343 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:54:11,343 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=63 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/C/5d659d756248488f8265d3fab5c18249 2023-03-29 03:54:11,344 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:11,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:11,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:11,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:11,345 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:11,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:11,346 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:11,346 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:54:11,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:11,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:11,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:11,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:11,348 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:54:11,348 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@61bc35c1[Count = 0] remaining members to acquire global barrier 2023-03-29 03:54:11,348 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:54:11,348 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:11,349 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/A/9c01fbdb1bda447ab51e87c017b39dd9 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/9c01fbdb1bda447ab51e87c017b39dd9 2023-03-29 03:54:11,349 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:11,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:11,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:11,350 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:54:11,350 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:54:11,350 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:54:11,350 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:11,350 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:54:11,352 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:11,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:11,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:11,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:11,352 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:54:11,352 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:11,352 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:54:11,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:11,354 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/9c01fbdb1bda447ab51e87c017b39dd9, entries=150, sequenceid=63, filesize=25.1 K 2023-03-29 03:54:11,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:11,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:11,355 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/B/7650c3e49e204614842a42aaff888b4c as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/7650c3e49e204614842a42aaff888b4c 2023-03-29 03:54:11,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:11,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:11,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:11,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:11,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:54:11,357 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:54:11,357 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:54:11,357 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:54:11,357 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:54:11,357 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:54:11,357 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:54:11,361 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/7650c3e49e204614842a42aaff888b4c, entries=150, sequenceid=63, filesize=11.6 K 2023-03-29 03:54:11,362 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/C/5d659d756248488f8265d3fab5c18249 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/5d659d756248488f8265d3fab5c18249 2023-03-29 03:54:11,363 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:11,363 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:11,363 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:11,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:11,363 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:11,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:11,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:11,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:11,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:11,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:11,364 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:11,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:11,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:11,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:11,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:11,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:11,366 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:11,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:11,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:11,368 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/5d659d756248488f8265d3fab5c18249, entries=150, sequenceid=63, filesize=11.6 K 2023-03-29 03:54:11,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:11,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:11,369 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=53.67 KB/54960 for ee0c8ff38ef29b37b45871dfc89897f0 in 146ms, sequenceid=63, compaction requested=true 2023-03-29 03:54:11,369 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ee0c8ff38ef29b37b45871dfc89897f0: 2023-03-29 03:54:11,369 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:54:11,369 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-29 03:54:11,369 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-29 03:54:11,369 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:54:11,369 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:54:11,370 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:11,370 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:11,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:11,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:11,370 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:11,370 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:54:11,370 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:11,370 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 77238 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-29 03:54:11,370 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-29 03:54:11,371 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1912): ee0c8ff38ef29b37b45871dfc89897f0/A is initiating minor compaction (all files) 2023-03-29 03:54:11,371 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1912): ee0c8ff38ef29b37b45871dfc89897f0/B is initiating minor compaction (all files) 2023-03-29 03:54:11,371 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:11,371 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ee0c8ff38ef29b37b45871dfc89897f0/B in TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. 2023-03-29 03:54:11,370 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:54:11,371 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/2c0c3cd702474babb481d56d8248c279, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/1b91a4cdcb154fb3ad48aa9f4dcd49c0, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/7650c3e49e204614842a42aaff888b4c] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp, totalSize=34.8 K 2023-03-29 03:54:11,370 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:11,371 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:54:11,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:11,371 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ee0c8ff38ef29b37b45871dfc89897f0/A in TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. 2023-03-29 03:54:11,371 DEBUG [Thread-985] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:54:11,371 DEBUG [Thread-985] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:54:11,371 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:11,371 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 2c0c3cd702474babb481d56d8248c279, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1680062050367 2023-03-29 03:54:11,371 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/a8fc006fc607439e884193ed31339f0d, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/310be1c32da64f1295d33cad9bb2c787, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/9c01fbdb1bda447ab51e87c017b39dd9] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp, totalSize=75.4 K 2023-03-29 03:54:11,371 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:11,372 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:11,372 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:11,372 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:11,372 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 1b91a4cdcb154fb3ad48aa9f4dcd49c0, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1680062050472 2023-03-29 03:54:11,372 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting a8fc006fc607439e884193ed31339f0d, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1680062050367 2023-03-29 03:54:11,373 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 7650c3e49e204614842a42aaff888b4c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=63, earliestPutTs=1680062051109 2023-03-29 03:54:11,373 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 310be1c32da64f1295d33cad9bb2c787, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1680062050472 2023-03-29 03:54:11,373 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ee0c8ff38ef29b37b45871dfc89897f0 3/3 column families, dataSize=60.38 KB heapSize=165.84 KB 2023-03-29 03:54:11,373 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee0c8ff38ef29b37b45871dfc89897f0, store=A 2023-03-29 03:54:11,373 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-29 03:54:11,374 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee0c8ff38ef29b37b45871dfc89897f0, store=B 2023-03-29 03:54:11,374 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-29 03:54:11,374 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee0c8ff38ef29b37b45871dfc89897f0, store=C 2023-03-29 03:54:11,374 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-29 03:54:11,374 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 9c01fbdb1bda447ab51e87c017b39dd9, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=63, earliestPutTs=1680062051109 2023-03-29 03:54:11,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on ee0c8ff38ef29b37b45871dfc89897f0 2023-03-29 03:54:11,395 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] throttle.PressureAwareThroughputController(145): ee0c8ff38ef29b37b45871dfc89897f0#B#compaction#135 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:54:11,414 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680062051404, max timestamp=1680062051412Num uniques 859; 2023-03-29 03:54:11,414 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-29 03:54:11,415 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680062051406, max timestamp=1680062051413Num uniques 859; 2023-03-29 03:54:11,415 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-29 03:54:11,416 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680062051406, max timestamp=1680062051413Num uniques 859; 2023-03-29 03:54:11,416 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-29 03:54:11,423 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] throttle.PressureAwareThroughputController(145): ee0c8ff38ef29b37b45871dfc89897f0#A#compaction#134 average throughput is 0.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:54:11,470 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:11,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40120 deadline: 1680062111461, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:11,471 DEBUG [Thread-985] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:54:11,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:54:11,474 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:11,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40148 deadline: 1680062111468, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:11,474 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:11,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40142 deadline: 1680062111470, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:11,485 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:54:11,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:54:11,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:54:11,497 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:54:11,497 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:11,497 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:54:11,497 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:54:11,497 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:11,498 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:11,504 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:11,504 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:11,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:11,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:11,504 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:11,504 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:54:11,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:11,505 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:11,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:54:11,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:11,505 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:54:11,505 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:54:11,506 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:54:11,506 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:54:11,506 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:11,506 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:54:11,508 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:54:11,508 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:54:11,508 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. 2023-03-29 03:54:11,508 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. started... 2023-03-29 03:54:11,508 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. as already flushing 2023-03-29 03:54:11,508 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. 2023-03-29 03:54:11,512 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:54:11,513 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:54:11,513 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:11,516 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:54:11,516 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:54:11,520 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:11,520 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:11,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:11,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:11,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:11,521 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:11,521 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:54:11,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:11,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:11,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:11,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:11,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:11,523 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:54:11,523 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@33e94f9a[Count = 0] remaining members to acquire global barrier 2023-03-29 03:54:11,523 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:54:11,524 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:11,525 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:11,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:11,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:11,525 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:54:11,525 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:54:11,525 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:54:11,525 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:11,525 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:54:11,526 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:54:11,527 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:11,527 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:11,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:11,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:11,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:11,527 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:54:11,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:11,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:11,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:11,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:11,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:11,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:11,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:11,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:54:11,530 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:54:11,530 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:54:11,530 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:54:11,530 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:54:11,530 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:54:11,530 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:54:11,534 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:11,534 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:11,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:11,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:11,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:11,534 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:11,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:11,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:11,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:11,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:11,534 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:11,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:11,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:11,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:11,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:11,535 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:11,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:11,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:11,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:11,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:11,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:11,538 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:11,538 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:11,538 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:11,538 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:11,538 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:11,538 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:54:11,538 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:54:11,538 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:54:11,538 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:11,538 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:11,538 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:11,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:11,539 DEBUG [Thread-985] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:54:11,539 DEBUG [Thread-985] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:54:11,539 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:11,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:11,539 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:11,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:11,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:11,574 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:11,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40120 deadline: 1680062111571, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:11,575 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:11,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40148 deadline: 1680062111575, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:11,576 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:11,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40142 deadline: 1680062111575, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:11,625 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:11,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40124 deadline: 1680062111625, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:11,630 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:11,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40112 deadline: 1680062111627, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:11,639 DEBUG [Thread-985] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:54:11,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:54:11,642 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:54:11,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:54:11,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:54:11,644 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:54:11,644 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:11,644 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:54:11,644 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:54:11,648 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:11,648 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:11,649 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:11,650 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:11,650 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:11,650 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:54:11,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:11,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:11,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:11,651 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:11,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:54:11,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:11,651 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:54:11,652 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:54:11,652 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:54:11,652 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:54:11,652 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:11,652 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:54:11,652 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:54:11,652 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:54:11,652 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. 2023-03-29 03:54:11,652 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. started... 2023-03-29 03:54:11,652 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. as already flushing 2023-03-29 03:54:11,652 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. 2023-03-29 03:54:11,652 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:54:11,652 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:54:11,652 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:11,652 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:54:11,652 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:54:11,654 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:11,654 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:11,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:11,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:11,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:11,654 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:11,654 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:54:11,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:11,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:11,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:11,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:11,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:11,657 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:54:11,657 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@400f2501[Count = 0] remaining members to acquire global barrier 2023-03-29 03:54:11,658 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:54:11,658 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:11,659 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:11,659 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:11,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:11,659 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:54:11,659 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:54:11,659 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:54:11,659 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:11,659 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:54:11,661 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:11,661 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:54:11,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:11,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:11,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:11,661 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:11,661 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:54:11,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:11,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:11,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:11,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:11,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:11,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:11,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:11,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:54:11,666 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:54:11,666 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:54:11,666 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:54:11,666 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:54:11,666 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:54:11,666 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:54:11,671 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:11,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:11,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:11,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:11,671 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:11,671 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:11,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:11,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:11,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:11,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:11,672 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:11,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:11,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:11,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:11,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:11,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:11,674 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:11,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:11,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:11,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:11,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:11,678 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:11,678 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:11,678 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:11,678 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:11,678 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:11,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:11,678 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:11,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:11,678 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:54:11,679 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:54:11,678 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:11,679 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:11,679 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:11,679 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:11,679 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:54:11,679 DEBUG [Thread-985] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:54:11,679 DEBUG [Thread-985] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:54:11,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:11,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:11,777 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:11,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40120 deadline: 1680062111776, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:11,777 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:11,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40148 deadline: 1680062111777, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:11,779 DEBUG [Thread-985] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:54:11,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:54:11,781 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:11,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40142 deadline: 1680062111778, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:11,782 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:54:11,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:54:11,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:54:11,791 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:54:11,791 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:11,791 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:54:11,791 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:54:11,791 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:11,791 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:11,793 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:11,793 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:11,793 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:11,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:11,793 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:11,793 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:54:11,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:11,794 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:11,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:54:11,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:11,794 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:54:11,795 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:54:11,795 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:54:11,795 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:54:11,795 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:11,795 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:54:11,800 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:54:11,800 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:54:11,800 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. 2023-03-29 03:54:11,800 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. started... 2023-03-29 03:54:11,800 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. as already flushing 2023-03-29 03:54:11,800 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. 2023-03-29 03:54:11,800 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:54:11,800 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:54:11,800 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:11,800 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:54:11,800 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:54:11,802 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:11,802 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:11,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:11,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:11,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:11,802 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:11,802 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:54:11,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:11,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:11,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:11,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:11,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:11,804 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:54:11,804 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3c7c070d[Count = 0] remaining members to acquire global barrier 2023-03-29 03:54:11,804 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:54:11,804 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:11,806 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:11,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:11,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:11,806 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:54:11,806 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:54:11,806 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:54:11,806 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:11,806 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:54:11,810 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:11,810 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:54:11,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:11,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:11,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:11,811 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:11,811 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:54:11,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:11,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:11,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:11,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:11,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:11,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:11,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:11,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:54:11,813 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:54:11,813 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:54:11,813 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:54:11,813 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:54:11,813 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:54:11,813 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:54:11,815 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:11,815 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:11,815 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:11,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:11,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:11,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:11,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:11,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:11,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:11,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:11,815 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:11,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:11,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:11,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:11,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:11,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:11,817 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:11,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:11,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:11,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:11,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:11,819 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:11,819 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:11,819 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:11,819 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:11,819 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:11,819 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:11,819 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:54:11,819 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:11,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:11,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:11,819 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:11,819 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:54:11,819 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:11,819 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:11,820 DEBUG [Thread-985] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:54:11,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:11,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:11,820 DEBUG [Thread-985] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:54:11,836 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/B/7c05e2107b844ff483fc71526dcdf5fc as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/7c05e2107b844ff483fc71526dcdf5fc 2023-03-29 03:54:11,842 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ee0c8ff38ef29b37b45871dfc89897f0/B of ee0c8ff38ef29b37b45871dfc89897f0 into 7c05e2107b844ff483fc71526dcdf5fc(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:54:11,842 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ee0c8ff38ef29b37b45871dfc89897f0: 2023-03-29 03:54:11,842 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0., storeName=ee0c8ff38ef29b37b45871dfc89897f0/B, priority=13, startTime=1680062051369; duration=0sec 2023-03-29 03:54:11,842 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:54:11,842 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-29 03:54:11,844 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-29 03:54:11,844 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1912): ee0c8ff38ef29b37b45871dfc89897f0/C is initiating minor compaction (all files) 2023-03-29 03:54:11,844 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ee0c8ff38ef29b37b45871dfc89897f0/C in TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. 2023-03-29 03:54:11,844 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/db71cfdcf2df4477b59fafaae3b40be2, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/c13f379d2c624f74bce0bc02627e55e0, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/5d659d756248488f8265d3fab5c18249] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp, totalSize=34.8 K 2023-03-29 03:54:11,845 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting db71cfdcf2df4477b59fafaae3b40be2, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1680062050367 2023-03-29 03:54:11,845 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting c13f379d2c624f74bce0bc02627e55e0, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1680062050472 2023-03-29 03:54:11,846 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 5d659d756248488f8265d3fab5c18249, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=63, earliestPutTs=1680062051109 2023-03-29 03:54:11,863 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] throttle.PressureAwareThroughputController(145): ee0c8ff38ef29b37b45871dfc89897f0#C#compaction#137 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:54:11,873 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032990bc2d9172534572a80d2c43dd21950e to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032990bc2d9172534572a80d2c43dd21950e 2023-03-29 03:54:11,919 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/A/2df21fa89e2847fdaa6e5c1978c4cf1e as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/2df21fa89e2847fdaa6e5c1978c4cf1e 2023-03-29 03:54:11,920 DEBUG [Thread-985] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:54:11,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:54:11,925 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ee0c8ff38ef29b37b45871dfc89897f0/A of ee0c8ff38ef29b37b45871dfc89897f0 into 2df21fa89e2847fdaa6e5c1978c4cf1e(size=25.2 K), total size for store is 25.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:54:11,925 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ee0c8ff38ef29b37b45871dfc89897f0: 2023-03-29 03:54:11,925 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0., storeName=ee0c8ff38ef29b37b45871dfc89897f0/A, priority=13, startTime=1680062051369; duration=0sec 2023-03-29 03:54:11,925 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:54:11,927 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:54:11,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:54:11,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:54:11,929 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:54:11,929 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:11,929 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:54:11,929 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:54:11,929 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:11,929 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:11,931 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:11,931 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:11,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:11,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:11,931 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:11,931 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:54:11,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:11,932 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:11,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:54:11,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:11,932 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:54:11,933 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:54:11,933 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:54:11,934 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:54:11,934 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:11,934 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:54:11,936 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:54:11,936 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:54:11,936 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. 2023-03-29 03:54:11,936 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. started... 2023-03-29 03:54:11,936 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. as already flushing 2023-03-29 03:54:11,936 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. 2023-03-29 03:54:11,936 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:54:11,936 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:54:11,936 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:11,936 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:54:11,936 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:54:11,939 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:11,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:11,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:11,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:11,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:11,940 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-03-29 03:54:11,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:11,941 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:11,941 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:11,941 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:54:11,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:11,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:11,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:11,946 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:54:11,946 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3cee3f7e[Count = 0] remaining members to acquire global barrier 2023-03-29 03:54:11,946 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:54:11,946 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:11,947 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:11,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:11,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:11,948 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:54:11,948 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:54:11,948 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:54:11,948 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:11,948 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:54:11,949 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:11,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:11,949 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:54:11,949 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:11,949 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:54:11,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:11,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:11,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:11,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:11,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:11,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:11,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:11,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:11,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:11,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:54:11,952 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:54:11,952 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:54:11,952 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:54:11,952 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:54:11,952 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:54:11,952 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:54:11,954 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:11,954 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:11,954 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:11,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:11,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:11,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:11,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:11,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:11,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:11,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:11,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:11,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:11,956 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:11,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:11,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:11,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:11,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:11,958 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:11,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:11,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:11,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:11,963 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:11,963 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:11,963 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:11,963 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:54:11,963 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:54:11,963 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:11,963 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:11,963 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:11,963 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:11,963 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:11,964 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:11,964 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:11,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:11,964 DEBUG [Thread-985] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:54:11,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:11,964 DEBUG [Thread-985] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:54:11,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:11,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:12,064 DEBUG [Thread-985] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:54:12,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:54:12,066 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:54:12,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:54:12,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:54:12,068 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:54:12,068 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:12,068 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:54:12,068 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:54:12,068 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:12,068 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:12,070 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:12,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:12,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:12,070 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:12,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:12,071 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:12,071 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:12,071 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:54:12,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:54:12,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:12,072 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:54:12,072 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:54:12,072 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:54:12,072 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:54:12,072 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:12,072 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:54:12,076 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:54:12,076 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:54:12,076 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. 2023-03-29 03:54:12,076 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. started... 2023-03-29 03:54:12,076 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. as already flushing 2023-03-29 03:54:12,076 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. 2023-03-29 03:54:12,076 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:54:12,076 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:54:12,076 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:12,076 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:54:12,076 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:54:12,078 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:12,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:12,078 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:12,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:12,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:12,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:12,079 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:12,079 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:54:12,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:12,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:12,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:12,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:12,082 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:12,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40120 deadline: 1680062112080, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:12,083 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:12,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40148 deadline: 1680062112081, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:12,084 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:54:12,084 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@458232e6[Count = 0] remaining members to acquire global barrier 2023-03-29 03:54:12,084 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:54:12,084 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:12,086 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:12,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:12,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:12,086 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:54:12,086 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:54:12,086 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:54:12,087 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:12,087 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:54:12,088 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:12,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40142 deadline: 1680062112084, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:12,093 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:12,093 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:54:12,093 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:12,093 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:54:12,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:12,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:12,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:12,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:12,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:12,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:12,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:12,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:12,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:12,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:12,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:54:12,100 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:54:12,100 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:54:12,100 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:54:12,100 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:54:12,100 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:54:12,100 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:54:12,102 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:12,102 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:12,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:12,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:12,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:12,102 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:12,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:12,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:12,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:12,103 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:12,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:12,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:12,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:12,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:12,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:12,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:12,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:12,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:12,105 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:12,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:12,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:12,108 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:12,108 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:54:12,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:12,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:12,108 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:12,108 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:54:12,108 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:12,109 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:12,109 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:12,109 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:12,109 DEBUG [Thread-985] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:54:12,108 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:12,109 DEBUG [Thread-985] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:54:12,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:12,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:12,109 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:12,109 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:12,109 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:12,209 DEBUG [Thread-985] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:54:12,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:54:12,212 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:54:12,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:54:12,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:54:12,214 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:54:12,214 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:12,214 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:54:12,214 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:54:12,215 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:12,215 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:12,216 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:12,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:12,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:12,217 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:12,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:12,217 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:12,217 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:54:12,217 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:12,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:54:12,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:12,218 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:54:12,219 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:54:12,219 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:54:12,219 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:54:12,219 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:12,219 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:54:12,219 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:54:12,219 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:54:12,219 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. 2023-03-29 03:54:12,219 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. started... 2023-03-29 03:54:12,219 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. as already flushing 2023-03-29 03:54:12,219 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. 2023-03-29 03:54:12,219 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:54:12,220 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:54:12,220 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:12,220 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:54:12,220 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:54:12,235 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:12,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:12,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:12,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:12,235 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:12,236 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:12,236 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:54:12,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:12,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:12,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:12,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:12,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:12,238 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:54:12,238 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c9c207a[Count = 0] remaining members to acquire global barrier 2023-03-29 03:54:12,238 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:54:12,238 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:12,240 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:12,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:12,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:12,241 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:54:12,241 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:54:12,241 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:54:12,241 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:12,241 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:54:12,243 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:12,243 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:54:12,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:12,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:12,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:12,243 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:12,243 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:54:12,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:12,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:12,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:12,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:12,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:12,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:12,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:12,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:54:12,248 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:54:12,248 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:54:12,249 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:54:12,249 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:54:12,249 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:54:12,249 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:54:12,251 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:12,251 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:12,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:12,251 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:12,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:12,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:12,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:12,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:12,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:12,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:12,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:12,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:12,253 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:12,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:12,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:12,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:12,254 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:12,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:12,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:12,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:12,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:12,257 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:12,258 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:12,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:12,258 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:12,258 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:12,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:12,258 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:54:12,258 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:54:12,258 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:12,258 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:12,258 DEBUG [Thread-985] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:54:12,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:12,258 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:12,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:12,258 DEBUG [Thread-985] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:54:12,258 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:12,259 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:12,259 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:12,341 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=89, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/A/9318058e9cb948d5b9aa3fd907a7ec82 2023-03-29 03:54:12,347 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/C/ab983ba6a9f2490bb7cb93a11d31aaae as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/ab983ba6a9f2490bb7cb93a11d31aaae 2023-03-29 03:54:12,352 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ee0c8ff38ef29b37b45871dfc89897f0/C of ee0c8ff38ef29b37b45871dfc89897f0 into ab983ba6a9f2490bb7cb93a11d31aaae(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:54:12,352 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ee0c8ff38ef29b37b45871dfc89897f0: 2023-03-29 03:54:12,352 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0., storeName=ee0c8ff38ef29b37b45871dfc89897f0/C, priority=13, startTime=1680062051369; duration=0sec 2023-03-29 03:54:12,352 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:54:12,359 DEBUG [Thread-985] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:54:12,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:54:12,361 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:54:12,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:54:12,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:54:12,366 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:54:12,366 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:12,366 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:54:12,366 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:54:12,366 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:12,366 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:12,369 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:12,369 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:12,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:12,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:12,369 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:12,369 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:54:12,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:12,370 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:12,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:54:12,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:12,370 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:54:12,370 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:54:12,370 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:54:12,371 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:12,371 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:54:12,371 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:54:12,371 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:54:12,371 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. 2023-03-29 03:54:12,371 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. started... 2023-03-29 03:54:12,371 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. as already flushing 2023-03-29 03:54:12,371 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. 2023-03-29 03:54:12,371 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:54:12,371 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:54:12,371 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:12,371 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:54:12,371 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:54:12,373 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:12,373 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:12,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:12,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:12,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:12,373 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:12,373 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:54:12,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:12,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:12,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:12,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:12,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:12,375 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:54:12,375 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2084148b[Count = 0] remaining members to acquire global barrier 2023-03-29 03:54:12,375 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:54:12,375 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:12,376 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:12,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:12,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:12,376 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:54:12,376 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:54:12,376 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:54:12,376 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:12,376 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:54:12,378 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:54:12,378 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:12,378 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:12,378 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:54:12,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:12,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:12,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:12,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:12,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:12,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:12,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:12,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:12,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:12,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:12,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:54:12,381 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:54:12,381 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:54:12,381 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:54:12,381 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:54:12,381 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:54:12,381 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:54:12,384 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:12,384 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:12,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:12,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:12,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:12,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:12,384 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:12,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:12,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:12,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:12,384 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:12,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:12,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:12,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:12,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:12,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:12,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:12,386 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:12,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:12,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:12,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:12,388 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:12,388 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:12,388 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:12,388 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:12,388 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:12,388 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:12,388 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:54:12,389 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:54:12,388 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:12,388 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:12,389 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:12,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:12,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:12,389 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:12,389 DEBUG [Thread-985] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:54:12,389 DEBUG [Thread-985] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:54:12,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:12,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:12,489 DEBUG [Thread-985] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:54:12,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:54:12,492 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:54:12,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:54:12,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:54:12,496 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:54:12,496 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:12,496 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:54:12,496 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:54:12,497 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:12,497 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:12,498 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:12,498 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:12,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:12,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:12,498 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:12,499 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:54:12,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:12,500 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:12,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:54:12,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:12,501 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:54:12,501 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:54:12,501 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:54:12,501 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:54:12,501 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:12,501 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:54:12,504 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:54:12,504 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:54:12,504 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. 2023-03-29 03:54:12,504 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. started... 2023-03-29 03:54:12,504 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. as already flushing 2023-03-29 03:54:12,504 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. 2023-03-29 03:54:12,504 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:54:12,504 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:54:12,504 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:12,504 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:54:12,504 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:54:12,506 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:12,506 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:12,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:12,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:12,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:12,507 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:12,507 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:54:12,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:12,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:12,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:12,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:12,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:12,510 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:54:12,510 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ddcda2a[Count = 0] remaining members to acquire global barrier 2023-03-29 03:54:12,510 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:54:12,510 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:12,514 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:12,514 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:12,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:12,515 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:12,515 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:54:12,515 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:54:12,515 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:54:12,515 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:54:12,518 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:12,518 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:54:12,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:12,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:12,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:12,518 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:12,518 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:54:12,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:12,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:12,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:12,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:12,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:12,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:12,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:12,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:54:12,524 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:54:12,524 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:54:12,524 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:54:12,524 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:54:12,524 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:54:12,524 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:54:12,527 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:12,527 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:12,527 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:12,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:12,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:12,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:12,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:12,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:12,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:12,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:12,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:12,528 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:12,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:12,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:12,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:12,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:12,530 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:12,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:12,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:12,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:12,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:12,533 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:12,533 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:12,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:12,534 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:12,534 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:12,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:12,534 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:12,534 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:54:12,534 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:54:12,534 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:12,534 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:54:12,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:12,534 DEBUG [Thread-985] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:54:12,534 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:12,534 DEBUG [Thread-985] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:54:12,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:12,534 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:12,534 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:12,535 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:12,589 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:12,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40148 deadline: 1680062112584, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:12,590 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:12,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40120 deadline: 1680062112586, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:12,590 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:12,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40142 deadline: 1680062112590, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:12,634 DEBUG [Thread-985] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:54:12,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:54:12,638 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:12,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40124 deadline: 1680062112636, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:12,639 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:12,639 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:54:12,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40112 deadline: 1680062112636, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:12,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:54:12,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:54:12,655 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:54:12,655 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:12,655 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:54:12,655 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:54:12,656 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:12,656 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:12,661 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:12,661 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:12,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:12,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:12,661 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:12,661 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:54:12,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:12,668 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:12,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:54:12,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:12,668 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:54:12,669 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:54:12,669 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:54:12,669 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:54:12,669 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:12,669 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:54:12,676 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:54:12,676 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:54:12,676 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. 2023-03-29 03:54:12,676 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. started... 2023-03-29 03:54:12,676 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. as already flushing 2023-03-29 03:54:12,676 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. 2023-03-29 03:54:12,676 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:54:12,676 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:54:12,676 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:12,676 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:54:12,676 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:54:12,679 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:12,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:12,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:12,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:12,679 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:12,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:12,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:12,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:12,680 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:12,681 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:54:12,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:12,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:12,682 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:54:12,682 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@391dd075[Count = 0] remaining members to acquire global barrier 2023-03-29 03:54:12,682 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:54:12,682 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:12,686 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:12,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:12,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:12,686 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:54:12,686 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:54:12,686 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:54:12,687 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:12,687 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:54:12,689 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:12,689 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:54:12,689 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:12,689 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:54:12,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:12,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:12,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:12,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:12,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:12,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:12,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:12,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:12,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:12,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:12,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:54:12,694 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:54:12,694 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:54:12,694 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:54:12,694 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:54:12,694 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:54:12,694 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:54:12,696 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:12,696 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:12,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:12,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:12,696 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:12,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:12,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:12,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:12,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:12,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:12,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:12,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:12,698 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:12,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:12,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:12,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:12,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:12,701 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:12,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:12,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:12,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:12,707 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:12,707 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:12,707 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:12,707 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:12,707 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:12,707 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:12,707 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:54:12,707 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:54:12,707 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:12,707 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:12,707 DEBUG [Thread-985] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:54:12,707 DEBUG [Thread-985] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:54:12,707 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:12,707 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:54:12,707 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:12,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:12,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:12,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:12,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:12,759 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=89 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/B/d948c26f8cd54373b4ec896ca7b375ef 2023-03-29 03:54:12,808 DEBUG [Thread-985] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:54:12,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:54:12,811 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:54:12,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:54:12,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:54:12,828 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:54:12,828 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:12,828 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:54:12,828 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:54:12,829 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:12,829 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:12,830 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:12,830 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:12,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:12,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:12,831 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:12,831 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:54:12,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:12,831 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:12,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:54:12,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:12,832 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:54:12,844 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:54:12,844 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:54:12,844 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:54:12,844 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:12,844 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:54:12,844 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=89 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/C/316e75c9b5c5445c925de3c23b28ae5a 2023-03-29 03:54:12,845 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:54:12,845 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:54:12,845 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. 2023-03-29 03:54:12,845 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. started... 2023-03-29 03:54:12,845 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. as already flushing 2023-03-29 03:54:12,845 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. 2023-03-29 03:54:12,845 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:54:12,845 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:54:12,845 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:12,845 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:54:12,845 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:54:12,848 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:12,848 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:12,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:12,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:12,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:12,848 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:12,848 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:54:12,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:12,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:12,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:12,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:12,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:12,852 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:54:12,852 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@249fb8dc[Count = 0] remaining members to acquire global barrier 2023-03-29 03:54:12,852 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:54:12,852 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:12,853 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:12,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:12,854 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/A/9318058e9cb948d5b9aa3fd907a7ec82 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/9318058e9cb948d5b9aa3fd907a7ec82 2023-03-29 03:54:12,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:12,854 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:54:12,854 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:54:12,854 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:54:12,855 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:12,855 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:54:12,856 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:12,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:12,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:12,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:12,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:12,856 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:54:12,856 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:12,856 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:54:12,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:12,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:12,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:12,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:12,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:12,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:12,860 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/9318058e9cb948d5b9aa3fd907a7ec82, entries=200, sequenceid=89, filesize=31.9 K 2023-03-29 03:54:12,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:54:12,861 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:54:12,861 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:54:12,861 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:54:12,861 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:54:12,861 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:54:12,861 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:54:12,861 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/B/d948c26f8cd54373b4ec896ca7b375ef as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/d948c26f8cd54373b4ec896ca7b375ef 2023-03-29 03:54:12,863 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:12,863 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:12,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:12,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:12,863 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:12,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:12,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:12,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:12,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:12,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:12,864 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:12,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:12,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:12,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:12,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:12,866 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:12,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:12,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:12,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:12,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:12,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:12,870 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:12,871 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:12,871 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:12,871 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:12,871 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:12,871 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:54:12,871 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:54:12,871 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:12,871 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:12,871 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:12,871 DEBUG [Thread-985] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:54:12,871 DEBUG [Thread-985] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:54:12,872 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:12,872 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:12,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:12,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:12,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:12,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:12,872 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:54:12,874 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/d948c26f8cd54373b4ec896ca7b375ef, entries=150, sequenceid=89, filesize=11.6 K 2023-03-29 03:54:12,876 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/C/316e75c9b5c5445c925de3c23b28ae5a as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/316e75c9b5c5445c925de3c23b28ae5a 2023-03-29 03:54:12,884 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/316e75c9b5c5445c925de3c23b28ae5a, entries=150, sequenceid=89, filesize=11.6 K 2023-03-29 03:54:12,886 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~165.80 KB/169776, currentSize=123.00 KB/125950 for ee0c8ff38ef29b37b45871dfc89897f0 in 1513ms, sequenceid=89, compaction requested=false 2023-03-29 03:54:12,886 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ee0c8ff38ef29b37b45871dfc89897f0: 2023-03-29 03:54:12,971 DEBUG [Thread-985] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:54:12,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:54:12,975 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:54:12,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:54:12,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:54:12,978 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:54:12,978 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:12,978 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:54:12,978 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:54:12,978 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:12,978 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:12,980 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:12,980 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:12,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:12,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:12,980 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:12,980 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:54:12,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:12,981 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:12,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:54:12,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:12,982 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:54:12,990 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:54:12,990 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:54:12,990 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:54:12,990 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:12,990 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:54:13,004 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:54:13,004 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:54:13,004 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. 2023-03-29 03:54:13,004 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. started... 2023-03-29 03:54:13,004 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ee0c8ff38ef29b37b45871dfc89897f0 3/3 column families, dataSize=123.00 KB heapSize=350.22 KB 2023-03-29 03:54:13,005 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee0c8ff38ef29b37b45871dfc89897f0, store=A 2023-03-29 03:54:13,005 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-29 03:54:13,005 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee0c8ff38ef29b37b45871dfc89897f0, store=B 2023-03-29 03:54:13,005 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-29 03:54:13,005 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee0c8ff38ef29b37b45871dfc89897f0, store=C 2023-03-29 03:54:13,005 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-29 03:54:13,078 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230329af83388d530e44c28e89f9f53a52ccec to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230329af83388d530e44c28e89f9f53a52ccec 2023-03-29 03:54:13,107 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=127, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/A/da2a464998bf47e9a6082c65e09a7fa1 2023-03-29 03:54:13,162 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=127 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/B/cb28c49a353e49f89ee3ad13ce3fc034 2023-03-29 03:54:13,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on ee0c8ff38ef29b37b45871dfc89897f0 2023-03-29 03:54:13,598 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. as already flushing 2023-03-29 03:54:13,611 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:13,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40148 deadline: 1680062113607, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:13,612 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:13,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40142 deadline: 1680062113608, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:13,613 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:13,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40120 deadline: 1680062113609, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:13,619 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=127 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/C/2939f52325594fd1b186471ff6fdc6b1 2023-03-29 03:54:13,624 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/A/da2a464998bf47e9a6082c65e09a7fa1 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/da2a464998bf47e9a6082c65e09a7fa1 2023-03-29 03:54:13,628 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/da2a464998bf47e9a6082c65e09a7fa1, entries=150, sequenceid=127, filesize=25.1 K 2023-03-29 03:54:13,629 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/B/cb28c49a353e49f89ee3ad13ce3fc034 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/cb28c49a353e49f89ee3ad13ce3fc034 2023-03-29 03:54:13,637 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/cb28c49a353e49f89ee3ad13ce3fc034, entries=150, sequenceid=127, filesize=11.6 K 2023-03-29 03:54:13,638 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/C/2939f52325594fd1b186471ff6fdc6b1 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/2939f52325594fd1b186471ff6fdc6b1 2023-03-29 03:54:13,643 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/2939f52325594fd1b186471ff6fdc6b1, entries=150, sequenceid=127, filesize=11.6 K 2023-03-29 03:54:13,644 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~123.00 KB/125950, heapSize ~350.17 KB/358576, currentSize=60.38 KB/61830 for ee0c8ff38ef29b37b45871dfc89897f0 in 640ms, sequenceid=127, compaction requested=true 2023-03-29 03:54:13,645 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ee0c8ff38ef29b37b45871dfc89897f0: 2023-03-29 03:54:13,645 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. 2023-03-29 03:54:13,645 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:54:13,645 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:54:13,645 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:13,648 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:54:13,649 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:54:13,650 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:13,650 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:13,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:13,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:13,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:13,651 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:13,651 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:54:13,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:13,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:13,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:13,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:13,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:13,653 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:54:13,653 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@243fd088[Count = 0] remaining members to acquire global barrier 2023-03-29 03:54:13,653 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:54:13,653 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:13,654 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:13,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:13,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:13,655 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:54:13,655 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:54:13,655 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:54:13,655 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:13,655 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:54:13,656 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:54:13,656 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:13,656 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:13,656 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:54:13,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:13,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:13,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:13,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:13,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:13,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:13,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:13,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:13,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:13,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:13,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:54:13,662 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:54:13,662 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:54:13,662 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:54:13,662 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:54:13,662 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:54:13,662 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:54:13,666 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:13,666 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:13,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:13,666 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:13,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:13,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:13,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:13,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:13,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:13,667 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:13,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:13,667 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:13,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:13,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:13,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:13,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:13,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:13,669 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:13,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:13,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:13,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:13,671 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:13,671 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:13,671 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:13,671 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:54:13,671 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:54:13,671 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:13,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:13,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:13,671 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:13,671 DEBUG [Thread-985] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:54:13,671 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:13,671 DEBUG [Thread-985] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:54:13,671 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:54:13,671 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:13,672 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:13,672 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:13,672 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:13,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:13,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:13,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on ee0c8ff38ef29b37b45871dfc89897f0 2023-03-29 03:54:13,715 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ee0c8ff38ef29b37b45871dfc89897f0 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-29 03:54:13,716 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee0c8ff38ef29b37b45871dfc89897f0, store=A 2023-03-29 03:54:13,716 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:54:13,716 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee0c8ff38ef29b37b45871dfc89897f0, store=B 2023-03-29 03:54:13,716 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:54:13,716 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee0c8ff38ef29b37b45871dfc89897f0, store=C 2023-03-29 03:54:13,716 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:54:13,731 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230329dbd47d51f45540ac9b38ac80e08db86a to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230329dbd47d51f45540ac9b38ac80e08db86a 2023-03-29 03:54:13,735 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:13,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40120 deadline: 1680062113734, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:13,735 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:13,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40148 deadline: 1680062113734, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:13,736 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:13,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40142 deadline: 1680062113735, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:13,772 DEBUG [Thread-985] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:54:13,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:54:13,774 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:54:13,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:54:13,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:54:13,776 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:54:13,776 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:13,776 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:54:13,776 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:54:13,776 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:13,776 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:13,777 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:13,777 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:13,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:13,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:13,778 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:13,778 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:54:13,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:13,778 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:13,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:54:13,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:13,779 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:54:13,779 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:54:13,779 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:54:13,779 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:54:13,779 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:13,779 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:54:13,780 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:54:13,780 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:54:13,780 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. 2023-03-29 03:54:13,780 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. started... 2023-03-29 03:54:13,780 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. as already flushing 2023-03-29 03:54:13,780 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. 2023-03-29 03:54:13,780 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:54:13,780 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:54:13,780 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:13,780 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:54:13,780 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:54:13,781 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:13,781 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:13,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:13,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:13,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:13,782 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:13,782 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:54:13,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:13,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:13,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:13,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:13,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:13,784 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:54:13,784 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f3943af[Count = 0] remaining members to acquire global barrier 2023-03-29 03:54:13,784 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:54:13,784 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:13,786 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:13,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:13,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:13,786 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:54:13,786 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:54:13,786 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:54:13,786 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:13,786 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:54:13,787 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:13,787 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:54:13,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:13,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:13,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:13,788 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:13,788 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:54:13,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:13,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:13,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:13,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:13,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:13,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:13,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:13,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:54:13,790 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:54:13,790 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:54:13,790 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:54:13,790 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:54:13,790 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:54:13,790 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:54:13,794 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:13,794 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:13,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:13,794 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:13,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:13,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:13,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:13,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:13,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:13,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:13,795 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:13,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:13,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:13,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:13,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:13,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:13,797 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:13,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:13,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:13,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:13,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:13,800 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:13,800 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:13,800 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:13,800 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:13,800 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:13,800 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:13,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:13,800 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:54:13,801 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:54:13,800 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:13,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:13,800 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:13,801 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:13,801 DEBUG [Thread-985] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:54:13,801 DEBUG [Thread-985] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:54:13,801 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:13,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:13,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:13,838 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:13,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40120 deadline: 1680062113836, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:13,838 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:13,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40148 deadline: 1680062113837, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:13,838 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:13,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40142 deadline: 1680062113837, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:13,901 DEBUG [Thread-985] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:54:13,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:54:13,903 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:54:13,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:54:13,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:54:13,905 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:54:13,905 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:13,905 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:54:13,905 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:54:13,905 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:13,905 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:13,906 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:13,906 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:13,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:13,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:13,907 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:13,907 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:54:13,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:13,907 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:13,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:54:13,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:13,907 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:54:13,908 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:54:13,908 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:54:13,908 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:54:13,908 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:13,908 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:54:13,908 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:54:13,908 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:54:13,908 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. 2023-03-29 03:54:13,908 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. started... 2023-03-29 03:54:13,908 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. as already flushing 2023-03-29 03:54:13,908 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. 2023-03-29 03:54:13,908 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:54:13,908 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:54:13,908 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:13,908 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:54:13,908 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:54:13,910 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:13,910 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:13,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:13,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:13,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:13,910 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:13,910 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:54:13,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:13,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:13,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:13,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:13,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:13,911 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:54:13,912 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7870b045[Count = 0] remaining members to acquire global barrier 2023-03-29 03:54:13,912 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:54:13,912 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:13,913 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:13,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:13,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:13,913 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:54:13,913 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:54:13,913 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:54:13,913 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:13,913 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:54:13,915 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:54:13,915 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:13,915 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:13,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:13,915 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:54:13,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:13,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:13,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:13,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:13,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:13,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:13,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:13,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:13,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:13,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:54:13,918 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:54:13,918 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:54:13,918 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:54:13,918 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:54:13,918 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:54:13,918 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:54:13,919 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:13,919 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:13,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:13,919 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:13,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:13,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:13,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:13,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:13,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:13,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:13,920 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:13,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:13,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:13,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:13,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:13,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:13,930 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:13,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:13,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:13,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:13,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:13,934 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:13,934 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:13,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:13,934 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:13,934 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:13,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:13,934 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:54:13,934 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:13,934 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:54:13,935 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:54:13,935 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:13,935 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:13,935 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:13,935 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:13,935 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:13,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:13,935 DEBUG [Thread-985] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:54:13,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:13,935 DEBUG [Thread-985] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:54:14,035 DEBUG [Thread-985] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:54:14,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:54:14,038 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:54:14,041 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:14,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:54:14,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40148 deadline: 1680062114039, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:14,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:54:14,041 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:54:14,041 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:14,041 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:54:14,042 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:54:14,042 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:14,042 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:14,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40142 deadline: 1680062114040, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:14,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40120 deadline: 1680062114040, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:14,044 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:14,044 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:14,045 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:14,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:14,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:14,045 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:14,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:14,046 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:14,046 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:54:14,047 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:14,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:54:14,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:14,047 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:54:14,047 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:54:14,047 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:54:14,047 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:54:14,048 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:14,048 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:54:14,048 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:54:14,048 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:54:14,048 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. 2023-03-29 03:54:14,048 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. started... 2023-03-29 03:54:14,048 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. as already flushing 2023-03-29 03:54:14,048 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. 2023-03-29 03:54:14,048 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:54:14,048 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:54:14,048 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:14,048 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:54:14,048 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:54:14,051 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:14,051 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:14,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:14,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:14,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:14,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:14,051 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:14,052 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:54:14,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:14,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:14,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:14,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:14,053 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:54:14,053 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@21f0becf[Count = 0] remaining members to acquire global barrier 2023-03-29 03:54:14,053 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:54:14,053 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:14,056 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:14,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:14,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:14,056 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:54:14,056 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:54:14,056 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:54:14,056 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:14,056 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:54:14,058 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:14,058 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:54:14,058 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:14,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:14,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:14,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:14,058 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:54:14,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:14,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:14,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:14,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:14,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:14,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:14,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:14,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:54:14,061 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:54:14,061 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:54:14,061 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:54:14,061 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:54:14,061 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:54:14,062 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:54:14,063 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:14,063 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:14,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:14,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:14,064 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:14,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:14,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:14,064 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:14,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:14,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:14,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:14,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:14,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:14,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:14,066 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:14,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:14,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:14,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:14,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:14,071 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:14,071 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:14,071 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:14,071 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:14,071 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:14,071 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:14,071 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:14,071 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:54:14,071 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:14,071 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:14,071 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:14,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:14,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:14,071 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:54:14,071 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:54:14,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:14,071 DEBUG [Thread-985] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:54:14,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:14,072 DEBUG [Thread-985] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:54:14,140 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=142, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/A/9beaaea60d37429d8490eeb284323256 2023-03-29 03:54:14,154 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=142 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/B/de48ee89f80a4f859f6c82f6e70bb8ae 2023-03-29 03:54:14,172 DEBUG [Thread-985] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:54:14,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:54:14,175 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=142 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/C/d1741c9d403a4407826e040016253bb3 2023-03-29 03:54:14,175 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:54:14,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:54:14,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:54:14,177 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:54:14,177 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:14,177 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:54:14,177 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:54:14,177 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:14,177 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:14,179 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:14,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:14,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:14,179 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:14,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:14,180 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:14,180 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:54:14,180 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:14,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:54:14,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:14,181 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:54:14,183 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/A/9beaaea60d37429d8490eeb284323256 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/9beaaea60d37429d8490eeb284323256 2023-03-29 03:54:14,184 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:54:14,184 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:54:14,184 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:54:14,184 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:14,184 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:54:14,184 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:54:14,184 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:54:14,185 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. 2023-03-29 03:54:14,185 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. started... 2023-03-29 03:54:14,185 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. as already flushing 2023-03-29 03:54:14,185 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. 2023-03-29 03:54:14,185 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:54:14,185 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:54:14,185 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:14,185 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:54:14,185 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:54:14,188 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:14,188 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:14,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:14,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:14,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:14,188 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:14,188 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:54:14,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:14,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:14,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:14,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:14,190 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/9beaaea60d37429d8490eeb284323256, entries=200, sequenceid=142, filesize=32.1 K 2023-03-29 03:54:14,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:14,192 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:54:14,192 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b1181e9[Count = 0] remaining members to acquire global barrier 2023-03-29 03:54:14,192 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:54:14,192 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:14,193 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:14,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:14,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:14,193 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:54:14,193 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:54:14,193 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:54:14,194 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:14,194 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:54:14,196 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:14,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:14,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:14,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:14,197 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:54:14,197 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:14,197 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:54:14,198 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/B/de48ee89f80a4f859f6c82f6e70bb8ae as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/de48ee89f80a4f859f6c82f6e70bb8ae 2023-03-29 03:54:14,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:14,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:14,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:14,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:14,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:14,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:14,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:14,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:54:14,205 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:54:14,205 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:54:14,205 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:54:14,205 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:54:14,205 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:54:14,205 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:54:14,206 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/de48ee89f80a4f859f6c82f6e70bb8ae, entries=150, sequenceid=142, filesize=11.8 K 2023-03-29 03:54:14,207 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/C/d1741c9d403a4407826e040016253bb3 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/d1741c9d403a4407826e040016253bb3 2023-03-29 03:54:14,208 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:14,208 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:14,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:14,208 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:14,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:14,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:14,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:14,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:14,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:14,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:14,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:14,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:14,210 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:14,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:14,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:14,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:14,212 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:14,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:14,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:14,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:14,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:14,213 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/d1741c9d403a4407826e040016253bb3, entries=150, sequenceid=142, filesize=11.8 K 2023-03-29 03:54:14,214 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for ee0c8ff38ef29b37b45871dfc89897f0 in 500ms, sequenceid=142, compaction requested=true 2023-03-29 03:54:14,214 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ee0c8ff38ef29b37b45871dfc89897f0: 2023-03-29 03:54:14,214 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:54:14,214 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-29 03:54:14,214 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:54:14,214 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-29 03:54:14,214 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-29 03:54:14,217 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 117167 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-29 03:54:14,217 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1912): ee0c8ff38ef29b37b45871dfc89897f0/A is initiating minor compaction (all files) 2023-03-29 03:54:14,217 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ee0c8ff38ef29b37b45871dfc89897f0/A in TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. 2023-03-29 03:54:14,217 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/2df21fa89e2847fdaa6e5c1978c4cf1e, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/9318058e9cb948d5b9aa3fd907a7ec82, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/da2a464998bf47e9a6082c65e09a7fa1, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/9beaaea60d37429d8490eeb284323256] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp, totalSize=114.4 K 2023-03-29 03:54:14,218 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:14,218 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:14,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:14,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:14,218 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:14,218 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:14,218 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:14,218 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47797 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-29 03:54:14,218 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:54:14,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:14,218 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1912): ee0c8ff38ef29b37b45871dfc89897f0/B is initiating minor compaction (all files) 2023-03-29 03:54:14,218 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ee0c8ff38ef29b37b45871dfc89897f0/B in TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. 2023-03-29 03:54:14,218 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/7c05e2107b844ff483fc71526dcdf5fc, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/d948c26f8cd54373b4ec896ca7b375ef, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/cb28c49a353e49f89ee3ad13ce3fc034, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/de48ee89f80a4f859f6c82f6e70bb8ae] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp, totalSize=46.7 K 2023-03-29 03:54:14,218 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:54:14,219 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:54:14,218 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:14,218 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:14,218 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 2df21fa89e2847fdaa6e5c1978c4cf1e, keycount=150, bloomtype=ROW, size=25.2 K, encoding=NONE, compression=NONE, seqNum=63, earliestPutTs=1680062051109 2023-03-29 03:54:14,219 DEBUG [Thread-985] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:54:14,219 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:14,219 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:14,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:14,219 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:14,219 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 7c05e2107b844ff483fc71526dcdf5fc, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=63, earliestPutTs=1680062051109 2023-03-29 03:54:14,219 DEBUG [Thread-985] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:54:14,219 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 9318058e9cb948d5b9aa3fd907a7ec82, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=89, earliestPutTs=1680062051366 2023-03-29 03:54:14,220 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting d948c26f8cd54373b4ec896ca7b375ef, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=89, earliestPutTs=1680062051366 2023-03-29 03:54:14,220 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting da2a464998bf47e9a6082c65e09a7fa1, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=127, earliestPutTs=1680062051427 2023-03-29 03:54:14,220 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting cb28c49a353e49f89ee3ad13ce3fc034, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=127, earliestPutTs=1680062051427 2023-03-29 03:54:14,220 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 9beaaea60d37429d8490eeb284323256, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=142, earliestPutTs=1680062053608 2023-03-29 03:54:14,221 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting de48ee89f80a4f859f6c82f6e70bb8ae, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=142, earliestPutTs=1680062053608 2023-03-29 03:54:14,229 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] throttle.PressureAwareThroughputController(145): ee0c8ff38ef29b37b45871dfc89897f0#B#compaction#147 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:54:14,233 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] throttle.PressureAwareThroughputController(145): ee0c8ff38ef29b37b45871dfc89897f0#A#compaction#146 average throughput is 3.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:54:14,319 DEBUG [Thread-985] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:54:14,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:54:14,322 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:54:14,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:54:14,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:54:14,324 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:54:14,324 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:14,324 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:54:14,324 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:54:14,324 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:14,324 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:14,327 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:14,327 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:14,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:14,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:14,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:14,328 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:14,328 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:54:14,328 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:14,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:54:14,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:14,329 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:54:14,329 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:54:14,329 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:54:14,329 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:54:14,329 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:14,329 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:54:14,331 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:54:14,331 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:54:14,331 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. 2023-03-29 03:54:14,331 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. started... 2023-03-29 03:54:14,331 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ee0c8ff38ef29b37b45871dfc89897f0 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-03-29 03:54:14,331 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee0c8ff38ef29b37b45871dfc89897f0, store=A 2023-03-29 03:54:14,331 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:54:14,331 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee0c8ff38ef29b37b45871dfc89897f0, store=B 2023-03-29 03:54:14,331 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:54:14,331 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee0c8ff38ef29b37b45871dfc89897f0, store=C 2023-03-29 03:54:14,332 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:54:14,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on ee0c8ff38ef29b37b45871dfc89897f0 2023-03-29 03:54:14,349 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. as already flushing 2023-03-29 03:54:14,354 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032979ca7dabb8224fa2aeb1239cff580b28 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032979ca7dabb8224fa2aeb1239cff580b28 2023-03-29 03:54:14,359 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=159, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/A/ae356b7a53184a01ba044c066a5c5acb 2023-03-29 03:54:14,366 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:14,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40120 deadline: 1680062114362, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:14,366 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:14,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40148 deadline: 1680062114363, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:14,367 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:14,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40142 deadline: 1680062114363, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:14,374 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=159 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/B/4b8e00de4e434030abb2a3de00c377ab 2023-03-29 03:54:14,403 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=159 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/C/a498ba292d1e40d6b0e1bc8086984e2d 2023-03-29 03:54:14,409 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/A/ae356b7a53184a01ba044c066a5c5acb as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/ae356b7a53184a01ba044c066a5c5acb 2023-03-29 03:54:14,414 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/ae356b7a53184a01ba044c066a5c5acb, entries=150, sequenceid=159, filesize=25.3 K 2023-03-29 03:54:14,416 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/B/4b8e00de4e434030abb2a3de00c377ab as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/4b8e00de4e434030abb2a3de00c377ab 2023-03-29 03:54:14,421 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/4b8e00de4e434030abb2a3de00c377ab, entries=150, sequenceid=159, filesize=11.8 K 2023-03-29 03:54:14,422 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/C/a498ba292d1e40d6b0e1bc8086984e2d as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/a498ba292d1e40d6b0e1bc8086984e2d 2023-03-29 03:54:14,428 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/a498ba292d1e40d6b0e1bc8086984e2d, entries=150, sequenceid=159, filesize=11.8 K 2023-03-29 03:54:14,429 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for ee0c8ff38ef29b37b45871dfc89897f0 in 98ms, sequenceid=159, compaction requested=true 2023-03-29 03:54:14,429 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ee0c8ff38ef29b37b45871dfc89897f0: 2023-03-29 03:54:14,429 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. 2023-03-29 03:54:14,429 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:54:14,429 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:54:14,429 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:14,430 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:54:14,430 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:54:14,431 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:14,431 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:14,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:14,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:14,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:14,432 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:14,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:14,432 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:54:14,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:14,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:14,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:14,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:14,433 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:54:14,433 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e50811f[Count = 0] remaining members to acquire global barrier 2023-03-29 03:54:14,433 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:54:14,433 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:14,434 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:14,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:14,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:14,435 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:54:14,435 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:54:14,435 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:54:14,435 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:14,435 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:54:14,436 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:14,436 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:54:14,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:14,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:14,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:14,436 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:14,437 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:54:14,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:14,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:14,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:14,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:14,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:14,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:14,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:14,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:54:14,439 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:54:14,439 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:54:14,439 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:54:14,439 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:54:14,439 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:54:14,439 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:54:14,442 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:14,442 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:14,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:14,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:14,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:14,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:14,442 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:14,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:14,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:14,442 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:14,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:14,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:14,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:14,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:14,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:14,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:14,444 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:14,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:14,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:14,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:14,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:14,446 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:14,446 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:14,446 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:14,446 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:14,446 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:14,446 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:14,446 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:54:14,446 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:54:14,446 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:54:14,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:14,446 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:14,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:14,446 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:14,446 DEBUG [Thread-985] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:54:14,447 DEBUG [Thread-985] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:54:14,446 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:14,447 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:14,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:14,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:14,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on ee0c8ff38ef29b37b45871dfc89897f0 2023-03-29 03:54:14,470 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ee0c8ff38ef29b37b45871dfc89897f0 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-03-29 03:54:14,472 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee0c8ff38ef29b37b45871dfc89897f0, store=A 2023-03-29 03:54:14,472 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:54:14,472 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee0c8ff38ef29b37b45871dfc89897f0, store=B 2023-03-29 03:54:14,472 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:54:14,472 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee0c8ff38ef29b37b45871dfc89897f0, store=C 2023-03-29 03:54:14,472 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:54:14,490 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:14,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40142 deadline: 1680062114487, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:14,491 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:14,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40120 deadline: 1680062114488, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:14,491 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:14,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40148 deadline: 1680062114489, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:14,547 DEBUG [Thread-985] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:54:14,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:54:14,549 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:54:14,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:54:14,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:54:14,550 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:54:14,550 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:14,550 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:54:14,550 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:54:14,551 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:14,551 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:14,552 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:14,552 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:14,552 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:14,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:14,552 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:14,552 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:54:14,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:14,553 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:14,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:54:14,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:14,553 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:54:14,553 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:54:14,553 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:54:14,553 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:54:14,554 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:14,554 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:54:14,554 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:54:14,554 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:54:14,554 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. 2023-03-29 03:54:14,554 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. started... 2023-03-29 03:54:14,554 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. as already flushing 2023-03-29 03:54:14,554 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. 2023-03-29 03:54:14,554 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:54:14,554 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:54:14,554 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:14,554 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:54:14,554 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:54:14,556 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:14,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:14,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:14,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:14,556 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:14,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:14,557 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:14,557 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:54:14,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:14,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:14,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:14,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:14,558 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:54:14,558 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@635ff808[Count = 0] remaining members to acquire global barrier 2023-03-29 03:54:14,558 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:54:14,558 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:14,560 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:14,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:14,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:14,560 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:54:14,560 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:54:14,560 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:54:14,560 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:14,560 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:54:14,562 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:14,562 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:54:14,562 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:14,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:14,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:14,562 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:54:14,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:14,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:14,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:14,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:14,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:14,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:14,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:14,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:14,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:54:14,564 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:54:14,564 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:54:14,564 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:54:14,564 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:54:14,564 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:54:14,564 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:54:14,566 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:14,566 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:14,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:14,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:14,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:14,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:14,566 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:14,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:14,566 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:14,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:14,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:14,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:14,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:14,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:14,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:14,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:14,568 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:14,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:14,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:14,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:14,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:14,570 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:14,570 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:14,570 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:14,570 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:54:14,570 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:54:14,570 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:14,570 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:14,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:14,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:14,570 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:54:14,570 DEBUG [Thread-985] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:54:14,571 DEBUG [Thread-985] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:54:14,570 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:14,570 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:14,571 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:14,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:14,571 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:14,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:14,571 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:14,594 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:14,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40142 deadline: 1680062114591, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:14,594 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:14,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40120 deadline: 1680062114592, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:14,594 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:14,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40148 deadline: 1680062114592, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:14,641 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/B/92b9a8817d904fa19b94c22ae40ccc44 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/92b9a8817d904fa19b94c22ae40ccc44 2023-03-29 03:54:14,648 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in ee0c8ff38ef29b37b45871dfc89897f0/B of ee0c8ff38ef29b37b45871dfc89897f0 into 92b9a8817d904fa19b94c22ae40ccc44(size=12.0 K), total size for store is 23.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:54:14,648 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ee0c8ff38ef29b37b45871dfc89897f0: 2023-03-29 03:54:14,649 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0., storeName=ee0c8ff38ef29b37b45871dfc89897f0/B, priority=12, startTime=1680062054214; duration=0sec 2023-03-29 03:54:14,649 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:54:14,649 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-29 03:54:14,651 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59833 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-29 03:54:14,651 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1912): ee0c8ff38ef29b37b45871dfc89897f0/C is initiating minor compaction (all files) 2023-03-29 03:54:14,651 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ee0c8ff38ef29b37b45871dfc89897f0/C in TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. 2023-03-29 03:54:14,651 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/ab983ba6a9f2490bb7cb93a11d31aaae, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/316e75c9b5c5445c925de3c23b28ae5a, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/2939f52325594fd1b186471ff6fdc6b1, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/d1741c9d403a4407826e040016253bb3, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/a498ba292d1e40d6b0e1bc8086984e2d] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp, totalSize=58.4 K 2023-03-29 03:54:14,651 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting ab983ba6a9f2490bb7cb93a11d31aaae, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=63, earliestPutTs=1680062051109 2023-03-29 03:54:14,652 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 316e75c9b5c5445c925de3c23b28ae5a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=89, earliestPutTs=1680062051366 2023-03-29 03:54:14,652 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:14,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40124 deadline: 1680062114648, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:14,652 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 2939f52325594fd1b186471ff6fdc6b1, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=127, earliestPutTs=1680062051427 2023-03-29 03:54:14,653 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting d1741c9d403a4407826e040016253bb3, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=142, earliestPutTs=1680062053608 2023-03-29 03:54:14,655 DEBUG [Thread-983] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4163 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0., hostname=jenkins-hbase4.apache.org,35599,1680062018534, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-29 03:54:14,655 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting a498ba292d1e40d6b0e1bc8086984e2d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=159, earliestPutTs=1680062053732 2023-03-29 03:54:14,657 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:14,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40112 deadline: 1680062114656, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:14,659 DEBUG [Thread-979] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4168 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0., hostname=jenkins-hbase4.apache.org,35599,1680062018534, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-29 03:54:14,666 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] throttle.PressureAwareThroughputController(145): ee0c8ff38ef29b37b45871dfc89897f0#C#compaction#152 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:54:14,671 DEBUG [Thread-985] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:54:14,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:54:14,675 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:54:14,676 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/C/0cd9cf0676354fc785b2c47b7596767f as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/0cd9cf0676354fc785b2c47b7596767f 2023-03-29 03:54:14,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:54:14,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:54:14,681 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:54:14,681 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:14,682 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:54:14,682 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:54:14,682 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:14,682 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:14,683 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in ee0c8ff38ef29b37b45871dfc89897f0/C of ee0c8ff38ef29b37b45871dfc89897f0 into 0cd9cf0676354fc785b2c47b7596767f(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:54:14,683 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ee0c8ff38ef29b37b45871dfc89897f0: 2023-03-29 03:54:14,683 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0., storeName=ee0c8ff38ef29b37b45871dfc89897f0/C, priority=11, startTime=1680062054214; duration=0sec 2023-03-29 03:54:14,683 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:54:14,684 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:14,684 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:14,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:14,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:14,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:14,685 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:14,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:54:14,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:14,685 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:54:14,688 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:54:14,688 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:54:14,688 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:54:14,688 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:14,688 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:54:14,692 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:14,692 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:54:14,692 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:54:14,692 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:54:14,692 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. 2023-03-29 03:54:14,692 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. started... 2023-03-29 03:54:14,692 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. as already flushing 2023-03-29 03:54:14,692 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. 2023-03-29 03:54:14,692 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:54:14,692 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:54:14,693 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:14,693 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:54:14,693 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:54:14,694 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:14,694 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:14,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:14,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:14,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:14,695 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:14,695 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:54:14,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:14,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:14,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:14,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:14,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:14,696 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:54:14,696 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@477d346a[Count = 0] remaining members to acquire global barrier 2023-03-29 03:54:14,696 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:54:14,696 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:14,698 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:14,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:14,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:14,698 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:54:14,698 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:54:14,698 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:54:14,698 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:14,698 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:54:14,699 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:14,699 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:54:14,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:14,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:14,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:14,699 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:14,699 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:54:14,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:14,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:14,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:14,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:14,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:14,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:14,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:14,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:54:14,702 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:54:14,702 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:54:14,702 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:54:14,702 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:54:14,702 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:54:14,702 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:54:14,705 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:14,705 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:14,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:14,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:14,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:14,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:14,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:14,705 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:14,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:14,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:14,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:14,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:14,706 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:14,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:14,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:14,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:14,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:14,710 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:14,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:14,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:14,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:14,715 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:14,715 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:14,715 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:14,715 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:14,715 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:14,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:14,715 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:14,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:14,715 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:54:14,715 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:54:14,715 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:54:14,715 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:14,716 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:14,716 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:14,716 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:14,716 DEBUG [Thread-985] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:54:14,716 DEBUG [Thread-985] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:54:14,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:14,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:14,798 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:14,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40120 deadline: 1680062114796, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:14,798 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:14,798 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:14,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40142 deadline: 1680062114796, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:14,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40148 deadline: 1680062114796, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:14,816 DEBUG [Thread-985] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:54:14,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:54:14,818 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:54:14,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:54:14,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:54:14,819 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:54:14,819 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:14,819 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:54:14,819 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:54:14,820 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:14,820 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:14,821 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:14,821 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:14,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:14,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:14,821 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:14,821 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:54:14,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:14,822 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:14,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:54:14,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:14,822 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:54:14,822 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:54:14,822 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:54:14,822 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:54:14,822 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:14,822 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:54:14,823 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:54:14,823 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:54:14,823 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. 2023-03-29 03:54:14,823 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. started... 2023-03-29 03:54:14,823 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. as already flushing 2023-03-29 03:54:14,823 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. 2023-03-29 03:54:14,823 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:54:14,823 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:54:14,823 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:14,823 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:54:14,823 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:54:14,826 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:14,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:14,826 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:14,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:14,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:14,826 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:14,826 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:54:14,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:14,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:14,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:14,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:14,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:14,828 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:54:14,828 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3014eba5[Count = 0] remaining members to acquire global barrier 2023-03-29 03:54:14,828 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:54:14,828 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:14,829 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:14,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:14,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:14,829 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:54:14,829 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:54:14,829 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:54:14,829 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:14,829 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:54:14,831 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:54:14,831 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:14,831 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:14,831 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:54:14,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:14,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:14,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:14,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:14,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:14,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:14,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:14,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:14,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:14,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:14,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:54:14,833 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:54:14,833 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:54:14,833 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:54:14,833 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:54:14,833 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:54:14,833 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:54:14,835 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:14,835 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:14,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:14,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:14,835 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:14,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:14,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:14,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:14,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:14,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:14,836 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:14,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:14,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:14,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:14,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:14,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:14,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:14,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:14,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:14,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:14,838 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:14,841 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:14,841 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:14,842 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:14,842 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:14,842 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:14,842 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:14,842 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:14,842 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:14,842 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:54:14,842 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:54:14,842 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:14,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:14,842 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:14,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:14,842 DEBUG [Thread-985] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:54:14,842 DEBUG [Thread-985] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:54:14,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:14,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:14,888 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303294bbf348ddcd9415fbcbb5264b4ceeb1f to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303294bbf348ddcd9415fbcbb5264b4ceeb1f 2023-03-29 03:54:14,942 DEBUG [Thread-985] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:54:14,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:54:14,947 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:54:14,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:54:14,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:54:14,948 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:54:14,948 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:14,948 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:54:14,948 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:54:14,949 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:14,949 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:14,950 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:14,950 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:14,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:14,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:14,951 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:14,951 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:54:14,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:14,951 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:14,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:54:14,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:14,952 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:54:14,960 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:54:14,960 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:54:14,960 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:54:14,960 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:14,960 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:54:14,961 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:54:14,961 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:54:14,961 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. 2023-03-29 03:54:14,961 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. started... 2023-03-29 03:54:14,961 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. as already flushing 2023-03-29 03:54:14,961 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. 2023-03-29 03:54:14,961 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:54:14,961 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:54:14,961 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:14,962 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:54:14,962 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:54:14,964 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:14,964 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:14,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:14,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:14,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:14,964 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:14,964 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:54:14,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:14,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:14,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:14,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:14,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:14,967 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:54:14,967 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2a56ccc1[Count = 0] remaining members to acquire global barrier 2023-03-29 03:54:14,967 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:54:14,967 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:14,969 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:14,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:14,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:14,969 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:14,969 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:54:14,969 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:54:14,969 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:54:14,969 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:54:14,972 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:14,972 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:54:14,972 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:14,972 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:54:14,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:14,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:14,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:14,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:14,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:14,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:14,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:14,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:14,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:14,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:14,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:54:14,976 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:54:14,976 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:54:14,976 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:54:14,976 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:54:14,976 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:54:14,976 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:54:14,982 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:14,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:14,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:14,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:14,983 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:14,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:14,983 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:14,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:14,983 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:14,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:14,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:14,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:14,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:14,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:14,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:14,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:14,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:14,985 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:14,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:14,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:14,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:14,989 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:14,989 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:14,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:14,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:14,990 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:14,990 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:14,990 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:14,990 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:14,990 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:14,990 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:14,990 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:14,994 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:14,994 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:54:14,995 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:54:14,995 DEBUG [Thread-985] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:54:14,995 DEBUG [Thread-985] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:54:14,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:14,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:15,059 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/A/e9f0af882d4846c99046a67a1818729b as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/e9f0af882d4846c99046a67a1818729b 2023-03-29 03:54:15,074 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in ee0c8ff38ef29b37b45871dfc89897f0/A of ee0c8ff38ef29b37b45871dfc89897f0 into e9f0af882d4846c99046a67a1818729b(size=25.5 K), total size for store is 50.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:54:15,074 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ee0c8ff38ef29b37b45871dfc89897f0: 2023-03-29 03:54:15,075 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0., storeName=ee0c8ff38ef29b37b45871dfc89897f0/A, priority=12, startTime=1680062054214; duration=0sec 2023-03-29 03:54:15,075 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:54:15,095 DEBUG [Thread-985] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:54:15,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:54:15,097 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:54:15,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:54:15,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:54:15,098 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:54:15,098 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:15,098 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:54:15,099 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:54:15,099 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:15,099 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:15,100 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:15,100 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:15,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:15,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:15,101 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:15,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:15,101 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:54:15,101 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:15,102 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:15,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:54:15,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:15,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40142 deadline: 1680062115099, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:15,102 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:54:15,102 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:15,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40120 deadline: 1680062115101, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:15,102 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:54:15,102 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:54:15,102 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:54:15,102 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:15,102 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:54:15,103 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:54:15,103 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:54:15,103 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. 2023-03-29 03:54:15,103 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. started... 2023-03-29 03:54:15,103 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. as already flushing 2023-03-29 03:54:15,103 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. 2023-03-29 03:54:15,103 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:54:15,103 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:54:15,103 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:15,103 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:54:15,103 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:54:15,103 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:15,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40148 deadline: 1680062115102, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:15,105 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:15,105 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:15,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:15,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:15,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:15,106 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:15,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:15,106 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:54:15,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:15,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:15,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:15,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:15,107 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:54:15,107 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3655ee3b[Count = 0] remaining members to acquire global barrier 2023-03-29 03:54:15,107 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:54:15,107 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:15,108 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:15,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:15,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:15,109 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:54:15,109 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:54:15,109 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:54:15,109 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:15,109 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:54:15,110 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:15,111 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:54:15,111 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:15,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:15,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:15,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:15,111 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:54:15,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:15,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:15,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:15,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:15,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:15,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:15,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:15,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:54:15,116 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:54:15,116 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:54:15,116 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:54:15,116 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:54:15,116 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:54:15,116 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:54:15,117 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:15,117 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:15,117 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:15,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:15,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:15,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:15,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:15,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:15,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:15,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:15,119 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:15,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:15,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:15,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:15,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:15,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:15,121 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:15,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:15,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:15,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:15,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:15,126 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:15,126 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:15,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:15,126 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:15,126 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:54:15,126 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:54:15,126 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:15,126 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:54:15,126 DEBUG [Thread-985] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:54:15,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:15,127 DEBUG [Thread-985] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:54:15,126 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:15,127 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:15,127 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:15,127 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:15,127 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:15,127 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:15,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:15,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:15,227 DEBUG [Thread-985] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:54:15,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:54:15,229 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:54:15,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:54:15,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:54:15,231 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:54:15,231 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:15,231 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:54:15,231 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:54:15,232 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:15,232 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:15,233 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:15,233 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:15,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:15,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:15,234 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:15,234 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:54:15,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:15,234 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:15,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:54:15,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:15,235 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:54:15,236 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:54:15,236 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:54:15,236 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:54:15,236 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:15,236 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:54:15,236 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:54:15,236 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:54:15,236 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. 2023-03-29 03:54:15,236 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. started... 2023-03-29 03:54:15,236 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. as already flushing 2023-03-29 03:54:15,236 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. 2023-03-29 03:54:15,236 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:54:15,236 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:54:15,236 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:15,236 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:54:15,236 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:54:15,240 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:15,240 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:15,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:15,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:15,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:15,240 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:15,240 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:54:15,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:15,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:15,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:15,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:15,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:15,242 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:54:15,242 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@182542ff[Count = 0] remaining members to acquire global barrier 2023-03-29 03:54:15,242 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:54:15,242 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:15,244 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:15,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:15,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:15,244 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:54:15,244 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:54:15,244 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:15,245 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:54:15,244 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:54:15,256 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:15,256 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:54:15,256 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:15,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:15,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:15,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:15,256 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:54:15,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:15,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:15,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:15,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:15,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:15,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:15,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:15,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:54:15,259 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:54:15,259 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:54:15,259 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:54:15,259 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:54:15,259 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:54:15,259 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:54:15,261 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:15,261 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:15,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:15,261 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:15,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:15,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:15,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:15,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:15,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:15,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:15,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:15,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:15,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:15,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:15,263 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:15,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:15,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:15,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:15,265 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:15,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:15,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:15,268 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:15,268 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:15,268 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:15,268 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:15,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:15,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:15,268 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:15,268 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:15,268 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:15,268 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:15,268 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:15,268 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:15,268 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:54:15,268 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:54:15,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:15,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:15,276 DEBUG [Thread-985] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:54:15,276 DEBUG [Thread-985] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:54:15,295 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=176, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/A/2bd37e8188f44cc49ca55220c82a7025 2023-03-29 03:54:15,319 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=176 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/B/71bcf77814a14be49fba329ee74aefb1 2023-03-29 03:54:15,371 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=176 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/C/176d6b10be3540cfa42110576e255a3d 2023-03-29 03:54:15,376 DEBUG [Thread-985] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:54:15,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:54:15,380 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:54:15,385 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/A/2bd37e8188f44cc49ca55220c82a7025 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/2bd37e8188f44cc49ca55220c82a7025 2023-03-29 03:54:15,391 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/2bd37e8188f44cc49ca55220c82a7025, entries=200, sequenceid=176, filesize=32.1 K 2023-03-29 03:54:15,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:54:15,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:54:15,392 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:54:15,392 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:15,392 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/B/71bcf77814a14be49fba329ee74aefb1 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/71bcf77814a14be49fba329ee74aefb1 2023-03-29 03:54:15,392 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:54:15,392 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:54:15,393 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:15,393 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:15,395 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:15,395 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:15,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:15,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:15,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:15,396 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:15,396 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:54:15,396 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:15,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:54:15,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:15,397 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:54:15,397 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:54:15,397 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:54:15,397 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:54:15,397 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:15,397 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:54:15,397 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:54:15,397 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:54:15,398 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. 2023-03-29 03:54:15,398 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. started... 2023-03-29 03:54:15,398 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. as already flushing 2023-03-29 03:54:15,398 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. 2023-03-29 03:54:15,398 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:54:15,398 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:54:15,398 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:15,398 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:54:15,398 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:54:15,400 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:15,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:15,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:15,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:15,400 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/71bcf77814a14be49fba329ee74aefb1, entries=150, sequenceid=176, filesize=11.8 K 2023-03-29 03:54:15,400 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:15,401 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/C/176d6b10be3540cfa42110576e255a3d as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/176d6b10be3540cfa42110576e255a3d 2023-03-29 03:54:15,402 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:15,402 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:54:15,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:15,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:15,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:15,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:15,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:15,404 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:54:15,404 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ffbcb9d[Count = 0] remaining members to acquire global barrier 2023-03-29 03:54:15,404 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:54:15,404 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:15,405 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:15,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:15,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:15,406 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:54:15,406 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:54:15,406 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:54:15,407 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/176d6b10be3540cfa42110576e255a3d, entries=150, sequenceid=176, filesize=11.8 K 2023-03-29 03:54:15,407 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:54:15,408 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:15,408 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:54:15,408 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:15,408 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for ee0c8ff38ef29b37b45871dfc89897f0 in 938ms, sequenceid=176, compaction requested=true 2023-03-29 03:54:15,408 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ee0c8ff38ef29b37b45871dfc89897f0: 2023-03-29 03:54:15,408 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:54:15,408 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:54:15,408 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-29 03:54:15,408 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-29 03:54:15,408 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:54:15,409 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:54:15,409 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:54:15,409 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:54:15,409 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:54:15,409 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:54:15,409 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:54:15,409 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:54:15,410 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 84918 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-29 03:54:15,410 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1912): ee0c8ff38ef29b37b45871dfc89897f0/A is initiating minor compaction (all files) 2023-03-29 03:54:15,411 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ee0c8ff38ef29b37b45871dfc89897f0/A in TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. 2023-03-29 03:54:15,411 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/e9f0af882d4846c99046a67a1818729b, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/ae356b7a53184a01ba044c066a5c5acb, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/2bd37e8188f44cc49ca55220c82a7025] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp, totalSize=82.9 K 2023-03-29 03:54:15,411 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36348 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-29 03:54:15,411 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1912): ee0c8ff38ef29b37b45871dfc89897f0/B is initiating minor compaction (all files) 2023-03-29 03:54:15,411 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ee0c8ff38ef29b37b45871dfc89897f0/B in TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. 2023-03-29 03:54:15,412 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/92b9a8817d904fa19b94c22ae40ccc44, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/4b8e00de4e434030abb2a3de00c377ab, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/71bcf77814a14be49fba329ee74aefb1] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp, totalSize=35.5 K 2023-03-29 03:54:15,412 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting e9f0af882d4846c99046a67a1818729b, keycount=150, bloomtype=ROW, size=25.5 K, encoding=NONE, compression=NONE, seqNum=142, earliestPutTs=1680062053608 2023-03-29 03:54:15,413 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 92b9a8817d904fa19b94c22ae40ccc44, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=142, earliestPutTs=1680062053608 2023-03-29 03:54:15,413 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting ae356b7a53184a01ba044c066a5c5acb, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=159, earliestPutTs=1680062053732 2023-03-29 03:54:15,414 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 4b8e00de4e434030abb2a3de00c377ab, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=159, earliestPutTs=1680062053732 2023-03-29 03:54:15,414 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 2bd37e8188f44cc49ca55220c82a7025, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=176, earliestPutTs=1680062054361 2023-03-29 03:54:15,415 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:15,415 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:15,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:15,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:15,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:15,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:15,415 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:15,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:15,415 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 71bcf77814a14be49fba329ee74aefb1, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=176, earliestPutTs=1680062054361 2023-03-29 03:54:15,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:15,417 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:15,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:15,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:15,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:15,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:15,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:15,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:15,418 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:15,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:15,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:15,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:15,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:15,423 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:15,423 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:15,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:15,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:15,423 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:15,423 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:15,423 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:15,423 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:15,423 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:15,423 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:15,423 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:15,423 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:54:15,423 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:54:15,423 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:15,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:15,424 DEBUG [Thread-985] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:54:15,424 DEBUG [Thread-985] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:54:15,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:15,424 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:54:15,433 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] throttle.PressureAwareThroughputController(145): ee0c8ff38ef29b37b45871dfc89897f0#B#compaction#155 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:54:15,440 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] throttle.PressureAwareThroughputController(145): ee0c8ff38ef29b37b45871dfc89897f0#A#compaction#156 average throughput is 2.46 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:54:15,524 DEBUG [Thread-985] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:54:15,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:54:15,527 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:54:15,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:54:15,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:54:15,528 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:54:15,528 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:15,528 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:54:15,528 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:54:15,528 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:15,528 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:15,530 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:15,531 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:15,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:15,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:15,531 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:15,531 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:54:15,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:15,531 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:15,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:54:15,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:15,532 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:54:15,536 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:54:15,536 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:54:15,536 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:54:15,536 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:15,536 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:54:15,536 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:54:15,536 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:54:15,536 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. 2023-03-29 03:54:15,536 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. started... 2023-03-29 03:54:15,536 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ee0c8ff38ef29b37b45871dfc89897f0 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-03-29 03:54:15,537 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee0c8ff38ef29b37b45871dfc89897f0, store=A 2023-03-29 03:54:15,537 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:54:15,537 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee0c8ff38ef29b37b45871dfc89897f0, store=B 2023-03-29 03:54:15,537 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:54:15,537 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee0c8ff38ef29b37b45871dfc89897f0, store=C 2023-03-29 03:54:15,537 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:54:15,562 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032951f58a685fe040ee839f6787143dc1a0 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032951f58a685fe040ee839f6787143dc1a0 2023-03-29 03:54:15,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on ee0c8ff38ef29b37b45871dfc89897f0 2023-03-29 03:54:15,605 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. as already flushing 2023-03-29 03:54:15,621 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:15,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40148 deadline: 1680062115618, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:15,622 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:15,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40142 deadline: 1680062115620, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:15,622 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:15,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40120 deadline: 1680062115621, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:15,724 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:15,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40148 deadline: 1680062115722, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:15,725 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:15,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40142 deadline: 1680062115723, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:15,725 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:15,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40120 deadline: 1680062115723, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:15,872 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/B/e61e6c7532674f6298e462f6f02afc5c as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/e61e6c7532674f6298e462f6f02afc5c 2023-03-29 03:54:15,881 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ee0c8ff38ef29b37b45871dfc89897f0/B of ee0c8ff38ef29b37b45871dfc89897f0 into e61e6c7532674f6298e462f6f02afc5c(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:54:15,881 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ee0c8ff38ef29b37b45871dfc89897f0: 2023-03-29 03:54:15,881 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0., storeName=ee0c8ff38ef29b37b45871dfc89897f0/B, priority=13, startTime=1680062055408; duration=0sec 2023-03-29 03:54:15,881 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:54:15,881 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-03-29 03:54:15,882 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-03-29 03:54:15,882 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-03-29 03:54:15,882 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. because compaction request was cancelled 2023-03-29 03:54:15,896 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/A/fda0e2948cf147e9a1efaa7733436dad as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/fda0e2948cf147e9a1efaa7733436dad 2023-03-29 03:54:15,909 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in ee0c8ff38ef29b37b45871dfc89897f0/A of ee0c8ff38ef29b37b45871dfc89897f0 into fda0e2948cf147e9a1efaa7733436dad(size=25.6 K), total size for store is 25.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:54:15,909 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ee0c8ff38ef29b37b45871dfc89897f0: 2023-03-29 03:54:15,909 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0., storeName=ee0c8ff38ef29b37b45871dfc89897f0/A, priority=13, startTime=1680062055408; duration=0sec 2023-03-29 03:54:15,909 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:54:15,927 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:15,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40120 deadline: 1680062115926, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:15,928 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:15,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40148 deadline: 1680062115926, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:15,929 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:15,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40142 deadline: 1680062115927, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:15,975 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=194, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/A/3f0ed081b874469e8224777187f9d70a 2023-03-29 03:54:16,231 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:16,231 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:16,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40148 deadline: 1680062116229, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:16,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40142 deadline: 1680062116230, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:16,231 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:16,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40120 deadline: 1680062116231, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:16,387 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=194 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/B/6e49a08930004a6096945b85cb0c9b78 2023-03-29 03:54:16,735 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:16,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40142 deadline: 1680062116733, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:16,737 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:16,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40148 deadline: 1680062116735, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:16,739 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:16,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40120 deadline: 1680062116736, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:16,808 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=194 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/C/445f38b2f5ed44feb05ef4041381382f 2023-03-29 03:54:16,814 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/A/3f0ed081b874469e8224777187f9d70a as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/3f0ed081b874469e8224777187f9d70a 2023-03-29 03:54:16,820 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/3f0ed081b874469e8224777187f9d70a, entries=150, sequenceid=194, filesize=25.3 K 2023-03-29 03:54:16,821 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/B/6e49a08930004a6096945b85cb0c9b78 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/6e49a08930004a6096945b85cb0c9b78 2023-03-29 03:54:16,829 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/6e49a08930004a6096945b85cb0c9b78, entries=150, sequenceid=194, filesize=11.8 K 2023-03-29 03:54:16,830 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/C/445f38b2f5ed44feb05ef4041381382f as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/445f38b2f5ed44feb05ef4041381382f 2023-03-29 03:54:16,837 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/445f38b2f5ed44feb05ef4041381382f, entries=150, sequenceid=194, filesize=11.8 K 2023-03-29 03:54:16,838 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for ee0c8ff38ef29b37b45871dfc89897f0 in 1302ms, sequenceid=194, compaction requested=true 2023-03-29 03:54:16,838 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ee0c8ff38ef29b37b45871dfc89897f0: 2023-03-29 03:54:16,838 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. 2023-03-29 03:54:16,838 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:54:16,838 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:54:16,838 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:16,838 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:54:16,838 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:54:16,840 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:16,841 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:16,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:16,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:16,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:16,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:16,842 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:16,842 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:54:16,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:16,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:16,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:16,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:16,846 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:54:16,846 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@13572e67[Count = 0] remaining members to acquire global barrier 2023-03-29 03:54:16,846 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:54:16,846 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:16,849 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:16,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:16,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:16,849 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:54:16,849 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:54:16,849 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:54:16,850 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:16,850 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:54:16,852 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:54:16,852 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:16,852 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:16,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:16,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:16,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:16,852 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:54:16,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:16,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:16,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:16,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:16,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:16,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:16,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:16,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:54:16,857 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:54:16,857 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:54:16,857 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:54:16,857 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:54:16,857 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:54:16,857 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:54:16,859 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:16,859 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:16,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:16,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:16,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:16,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:16,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:16,859 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:16,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:16,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:16,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:16,861 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:16,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:16,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:16,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:16,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:16,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:16,863 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:16,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:16,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:16,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:16,866 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:16,866 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:16,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:16,866 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:54:16,866 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:54:16,866 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:54:16,866 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:16,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:16,866 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:16,866 DEBUG [Thread-985] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:54:16,866 DEBUG [Thread-985] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:54:16,866 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:16,866 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:16,866 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:16,866 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:16,866 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:16,866 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:16,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:16,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:16,966 DEBUG [Thread-985] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:54:16,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:54:16,969 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:54:16,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:54:16,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:54:16,970 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:54:16,970 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:16,970 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:54:16,970 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:54:16,970 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:16,970 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:16,977 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:16,977 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:16,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:16,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:16,978 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:16,978 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:54:16,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:16,978 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:16,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:54:16,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:16,979 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:54:16,979 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:54:16,979 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:54:16,979 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:54:16,979 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:16,979 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:54:16,979 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:54:16,979 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:54:16,979 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. 2023-03-29 03:54:16,979 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. started... 2023-03-29 03:54:16,980 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ee0c8ff38ef29b37b45871dfc89897f0 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-03-29 03:54:16,980 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee0c8ff38ef29b37b45871dfc89897f0, store=A 2023-03-29 03:54:16,980 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:54:16,980 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee0c8ff38ef29b37b45871dfc89897f0, store=B 2023-03-29 03:54:16,980 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:54:16,980 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee0c8ff38ef29b37b45871dfc89897f0, store=C 2023-03-29 03:54:16,980 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:54:17,005 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230329acb2a65e0a5e42e5814e41e602bbf68b to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230329acb2a65e0a5e42e5814e41e602bbf68b 2023-03-29 03:54:17,423 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=213, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/A/0d5a970b176e497dbf41a2abbce4c4d4 2023-03-29 03:54:17,445 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=213 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/B/e7627510cac2403ab1578315b8034ee5 2023-03-29 03:54:17,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on ee0c8ff38ef29b37b45871dfc89897f0 2023-03-29 03:54:17,740 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. as already flushing 2023-03-29 03:54:17,753 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:17,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40142 deadline: 1680062117752, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:17,753 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:17,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40148 deadline: 1680062117753, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:17,755 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:17,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40120 deadline: 1680062117753, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:17,855 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:17,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40142 deadline: 1680062117854, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:17,856 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:17,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40148 deadline: 1680062117854, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:17,858 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:17,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40120 deadline: 1680062117856, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:17,861 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=213 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/C/49676f4e28d94dd69ac114f618221c21 2023-03-29 03:54:17,867 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/A/0d5a970b176e497dbf41a2abbce4c4d4 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/0d5a970b176e497dbf41a2abbce4c4d4 2023-03-29 03:54:17,872 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/0d5a970b176e497dbf41a2abbce4c4d4, entries=150, sequenceid=213, filesize=25.3 K 2023-03-29 03:54:17,873 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/B/e7627510cac2403ab1578315b8034ee5 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/e7627510cac2403ab1578315b8034ee5 2023-03-29 03:54:17,888 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/e7627510cac2403ab1578315b8034ee5, entries=150, sequenceid=213, filesize=11.8 K 2023-03-29 03:54:17,891 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/C/49676f4e28d94dd69ac114f618221c21 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/49676f4e28d94dd69ac114f618221c21 2023-03-29 03:54:17,896 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/49676f4e28d94dd69ac114f618221c21, entries=150, sequenceid=213, filesize=11.8 K 2023-03-29 03:54:17,896 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for ee0c8ff38ef29b37b45871dfc89897f0 in 916ms, sequenceid=213, compaction requested=true 2023-03-29 03:54:17,897 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ee0c8ff38ef29b37b45871dfc89897f0: 2023-03-29 03:54:17,897 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. 2023-03-29 03:54:17,897 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:54:17,897 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:54:17,897 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:17,897 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:54:17,897 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:54:17,899 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:17,899 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:17,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:17,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:17,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:17,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:17,899 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:17,899 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:54:17,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:17,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:17,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:17,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:17,901 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:54:17,901 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a6a3e00[Count = 0] remaining members to acquire global barrier 2023-03-29 03:54:17,901 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:54:17,901 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:17,907 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:17,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:17,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:17,907 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:54:17,908 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:54:17,907 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:17,908 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:54:17,908 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:54:17,910 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:54:17,910 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:17,910 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:54:17,910 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:17,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:17,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:17,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:17,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:17,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:17,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:17,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:17,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:17,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:17,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:17,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:54:17,913 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:54:17,913 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:54:17,913 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:54:17,913 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:54:17,913 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:54:17,913 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:54:17,915 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:17,915 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:17,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:17,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:17,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:17,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:17,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:17,915 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:17,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:17,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:17,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:17,916 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:17,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:17,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:17,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:17,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:17,917 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:17,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:17,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:17,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:17,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:17,920 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:17,920 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:17,920 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:17,920 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:17,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:17,920 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:17,920 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:54:17,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:17,920 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:54:17,920 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:54:17,920 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:17,920 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:17,920 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:17,920 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:17,921 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:17,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:17,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:17,921 DEBUG [Thread-985] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:54:17,921 DEBUG [Thread-985] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:54:18,021 DEBUG [Thread-985] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:54:18,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:54:18,024 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:54:18,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:54:18,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:54:18,026 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:54:18,026 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:18,026 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:54:18,026 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:54:18,026 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:18,026 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:18,028 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:18,028 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:18,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:18,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:18,028 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:18,029 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:54:18,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:18,029 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:18,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:54:18,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:18,030 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:54:18,036 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:54:18,036 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:54:18,036 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:54:18,036 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:18,036 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:54:18,036 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:54:18,037 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:54:18,037 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. 2023-03-29 03:54:18,037 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. started... 2023-03-29 03:54:18,037 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ee0c8ff38ef29b37b45871dfc89897f0 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-03-29 03:54:18,037 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee0c8ff38ef29b37b45871dfc89897f0, store=A 2023-03-29 03:54:18,037 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:54:18,037 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee0c8ff38ef29b37b45871dfc89897f0, store=B 2023-03-29 03:54:18,037 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:54:18,037 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee0c8ff38ef29b37b45871dfc89897f0, store=C 2023-03-29 03:54:18,037 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:54:18,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on ee0c8ff38ef29b37b45871dfc89897f0 2023-03-29 03:54:18,061 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. as already flushing 2023-03-29 03:54:18,081 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:18,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40120 deadline: 1680062118076, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:18,082 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:18,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40142 deadline: 1680062118077, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:18,082 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:18,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40148 deadline: 1680062118080, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:18,184 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:18,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40120 deadline: 1680062118183, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:18,184 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:18,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40142 deadline: 1680062118183, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:18,184 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:18,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40148 deadline: 1680062118183, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:18,388 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:18,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40120 deadline: 1680062118385, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:18,388 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:18,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40148 deadline: 1680062118386, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:18,388 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:18,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40142 deadline: 1680062118386, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:18,466 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303299a24b2a7a4244516a6426c9b489534b5 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303299a24b2a7a4244516a6426c9b489534b5 2023-03-29 03:54:18,471 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=228, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/A/7ecb6ce9ba7941ba80c3bed0abe1079d 2023-03-29 03:54:18,661 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:18,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40112 deadline: 1680062118661, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:18,663 DEBUG [Thread-979] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8172 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0., hostname=jenkins-hbase4.apache.org,35599,1680062018534, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-29 03:54:18,691 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:18,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40148 deadline: 1680062118689, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:18,692 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:18,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40142 deadline: 1680062118690, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:18,692 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:18,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40120 deadline: 1680062118691, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:18,695 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:18,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40124 deadline: 1680062118694, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:18,697 DEBUG [Thread-983] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8204 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0., hostname=jenkins-hbase4.apache.org,35599,1680062018534, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-29 03:54:18,882 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=228 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/B/eb60a436a7fc4f439a5c78a68210c89f 2023-03-29 03:54:18,896 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=228 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/C/5a497fa220894f30bf9513d7734546c7 2023-03-29 03:54:18,902 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/A/7ecb6ce9ba7941ba80c3bed0abe1079d as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/7ecb6ce9ba7941ba80c3bed0abe1079d 2023-03-29 03:54:18,907 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/7ecb6ce9ba7941ba80c3bed0abe1079d, entries=150, sequenceid=228, filesize=25.3 K 2023-03-29 03:54:18,909 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/B/eb60a436a7fc4f439a5c78a68210c89f as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/eb60a436a7fc4f439a5c78a68210c89f 2023-03-29 03:54:18,913 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/eb60a436a7fc4f439a5c78a68210c89f, entries=150, sequenceid=228, filesize=11.8 K 2023-03-29 03:54:18,915 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/C/5a497fa220894f30bf9513d7734546c7 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/5a497fa220894f30bf9513d7734546c7 2023-03-29 03:54:18,921 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/5a497fa220894f30bf9513d7734546c7, entries=150, sequenceid=228, filesize=11.8 K 2023-03-29 03:54:18,922 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for ee0c8ff38ef29b37b45871dfc89897f0 in 885ms, sequenceid=228, compaction requested=true 2023-03-29 03:54:18,922 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ee0c8ff38ef29b37b45871dfc89897f0: 2023-03-29 03:54:18,922 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. 2023-03-29 03:54:18,922 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:54:18,922 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:54:18,922 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:18,922 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:54:18,922 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:54:18,924 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:18,924 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:18,924 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:18,925 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:54:18,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:18,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:18,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:18,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:18,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:18,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:18,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:18,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:18,927 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:54:18,927 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@386ab708[Count = 0] remaining members to acquire global barrier 2023-03-29 03:54:18,927 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:54:18,927 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:18,929 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:18,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:18,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:18,929 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:54:18,929 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:54:18,929 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:54:18,929 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:18,929 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:54:18,930 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:18,930 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:54:18,930 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:18,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:18,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:18,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:18,930 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:54:18,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:18,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:18,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:18,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:18,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:18,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:18,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:18,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:54:18,934 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:54:18,934 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:54:18,934 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:54:18,934 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:54:18,935 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:54:18,935 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:54:18,936 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:18,936 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:18,936 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:18,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:18,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:18,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:18,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:18,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:18,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:18,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:18,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:18,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:18,937 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:18,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:18,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:18,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:18,939 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:18,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:18,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:18,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:18,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:18,941 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:18,941 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:18,941 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:18,941 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:54:18,941 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:54:18,941 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:54:18,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:18,942 DEBUG [Thread-985] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:54:18,942 DEBUG [Thread-985] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:54:18,941 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:18,941 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:18,942 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:18,942 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:18,942 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:18,942 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:18,942 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:18,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:18,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:18,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:19,042 DEBUG [Thread-985] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:54:19,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:54:19,044 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:54:19,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:54:19,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:54:19,046 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:54:19,046 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:19,046 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:54:19,046 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:54:19,046 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:19,046 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:19,049 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:19,049 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:19,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:19,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:19,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:19,049 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:19,050 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:54:19,050 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:19,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:54:19,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:19,050 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:54:19,050 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:54:19,051 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:54:19,051 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:54:19,051 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:19,051 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:54:19,051 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:54:19,051 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:54:19,051 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. 2023-03-29 03:54:19,051 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. started... 2023-03-29 03:54:19,051 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ee0c8ff38ef29b37b45871dfc89897f0 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-03-29 03:54:19,052 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee0c8ff38ef29b37b45871dfc89897f0, store=A 2023-03-29 03:54:19,052 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:54:19,052 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee0c8ff38ef29b37b45871dfc89897f0, store=B 2023-03-29 03:54:19,052 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:54:19,052 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee0c8ff38ef29b37b45871dfc89897f0, store=C 2023-03-29 03:54:19,052 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:54:19,080 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303299cfc87fe4d244599a37c9d969df565fe to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303299cfc87fe4d244599a37c9d969df565fe 2023-03-29 03:54:19,085 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=245, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/A/3110413ad6634b3e8578f2e900123436 2023-03-29 03:54:19,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on ee0c8ff38ef29b37b45871dfc89897f0 2023-03-29 03:54:19,196 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. as already flushing 2023-03-29 03:54:19,210 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:19,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40120 deadline: 1680062119208, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:19,212 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:19,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40142 deadline: 1680062119210, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:19,213 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:19,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40148 deadline: 1680062119210, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:19,314 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:19,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40120 deadline: 1680062119311, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:19,315 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:19,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40142 deadline: 1680062119314, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:19,315 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:19,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40148 deadline: 1680062119314, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:19,503 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=245 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/B/20a663170f70472599a8d4c8e072398a 2023-03-29 03:54:19,519 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:19,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40120 deadline: 1680062119517, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:19,519 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:19,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40148 deadline: 1680062119517, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:19,520 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:19,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40142 deadline: 1680062119517, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:19,821 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:19,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40120 deadline: 1680062119820, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:19,822 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:19,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40148 deadline: 1680062119820, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:19,824 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:19,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40142 deadline: 1680062119822, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:19,914 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=245 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/C/26be1f3657ce4301bc411a7166777796 2023-03-29 03:54:19,919 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/A/3110413ad6634b3e8578f2e900123436 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/3110413ad6634b3e8578f2e900123436 2023-03-29 03:54:19,924 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/3110413ad6634b3e8578f2e900123436, entries=150, sequenceid=245, filesize=25.3 K 2023-03-29 03:54:19,925 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/B/20a663170f70472599a8d4c8e072398a as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/20a663170f70472599a8d4c8e072398a 2023-03-29 03:54:19,934 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/20a663170f70472599a8d4c8e072398a, entries=150, sequenceid=245, filesize=11.8 K 2023-03-29 03:54:19,935 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/C/26be1f3657ce4301bc411a7166777796 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/26be1f3657ce4301bc411a7166777796 2023-03-29 03:54:19,939 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/26be1f3657ce4301bc411a7166777796, entries=150, sequenceid=245, filesize=11.8 K 2023-03-29 03:54:19,940 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for ee0c8ff38ef29b37b45871dfc89897f0 in 889ms, sequenceid=245, compaction requested=true 2023-03-29 03:54:19,940 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ee0c8ff38ef29b37b45871dfc89897f0: 2023-03-29 03:54:19,940 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. 2023-03-29 03:54:19,940 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:54:19,940 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:54:19,940 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:19,940 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:54:19,940 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:54:19,942 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:19,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:19,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:19,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:19,942 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:19,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:19,943 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:19,943 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:54:19,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:19,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:19,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:19,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:19,944 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:54:19,944 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@699c34bb[Count = 0] remaining members to acquire global barrier 2023-03-29 03:54:19,944 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:54:19,944 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:19,945 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:19,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:19,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:19,945 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:54:19,945 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:54:19,945 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:54:19,945 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:19,945 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:54:19,948 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:19,948 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:54:19,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:19,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:19,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:19,948 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:19,948 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:54:19,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:19,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:19,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:19,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:19,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:19,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:19,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:19,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:54:19,951 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:54:19,951 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:54:19,951 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:54:19,952 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:54:19,952 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:54:19,952 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:54:19,953 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:19,953 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:19,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:19,953 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:19,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:19,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:19,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:19,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:19,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:19,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:19,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:19,954 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:19,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:19,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:19,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:19,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:19,955 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:19,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:19,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:19,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:19,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:19,957 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:19,957 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:19,957 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:19,957 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:19,957 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:19,957 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:54:19,957 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:54:19,957 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:19,957 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:19,957 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:19,957 DEBUG [Thread-985] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:54:19,957 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:19,957 DEBUG [Thread-985] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:54:19,957 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:54:19,957 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:19,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:19,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:19,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:19,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:20,057 DEBUG [Thread-985] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:54:20,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:54:20,060 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:54:20,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:54:20,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:54:20,061 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:54:20,061 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:20,061 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:54:20,061 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:54:20,062 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:20,062 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:20,063 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:20,063 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:20,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:20,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:20,063 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:20,063 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:54:20,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:20,064 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:20,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:54:20,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:20,064 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:54:20,064 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:54:20,065 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:54:20,065 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:54:20,065 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:20,065 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:54:20,065 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:54:20,065 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:54:20,065 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. 2023-03-29 03:54:20,065 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. started... 2023-03-29 03:54:20,065 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ee0c8ff38ef29b37b45871dfc89897f0 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-03-29 03:54:20,065 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee0c8ff38ef29b37b45871dfc89897f0, store=A 2023-03-29 03:54:20,066 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:54:20,066 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee0c8ff38ef29b37b45871dfc89897f0, store=B 2023-03-29 03:54:20,066 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:54:20,066 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee0c8ff38ef29b37b45871dfc89897f0, store=C 2023-03-29 03:54:20,066 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:54:20,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on ee0c8ff38ef29b37b45871dfc89897f0 2023-03-29 03:54:20,327 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. as already flushing 2023-03-29 03:54:20,350 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:20,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40120 deadline: 1680062120348, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:20,351 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:20,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40142 deadline: 1680062120349, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:20,352 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:20,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40148 deadline: 1680062120350, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:20,452 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:20,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40120 deadline: 1680062120451, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:20,454 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:20,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40142 deadline: 1680062120453, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:20,454 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:20,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40148 deadline: 1680062120453, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:20,483 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230329e48d090c005f40b68d320494d16ccbcf to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230329e48d090c005f40b68d320494d16ccbcf 2023-03-29 03:54:20,495 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=260, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/A/85008277797347258066bb667b5c6765 2023-03-29 03:54:20,518 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=260 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/B/0bd911f13f094a62bbedd35e6faf71a0 2023-03-29 03:54:20,655 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:20,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40120 deadline: 1680062120654, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:20,657 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:20,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40148 deadline: 1680062120655, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:20,657 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:20,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40142 deadline: 1680062120656, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:20,934 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=260 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/C/362fbe53edaf44c8acf6a621729edb1f 2023-03-29 03:54:20,939 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/A/85008277797347258066bb667b5c6765 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/85008277797347258066bb667b5c6765 2023-03-29 03:54:20,943 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/85008277797347258066bb667b5c6765, entries=150, sequenceid=260, filesize=25.4 K 2023-03-29 03:54:20,944 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/B/0bd911f13f094a62bbedd35e6faf71a0 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/0bd911f13f094a62bbedd35e6faf71a0 2023-03-29 03:54:20,948 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/0bd911f13f094a62bbedd35e6faf71a0, entries=150, sequenceid=260, filesize=11.9 K 2023-03-29 03:54:20,949 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/C/362fbe53edaf44c8acf6a621729edb1f as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/362fbe53edaf44c8acf6a621729edb1f 2023-03-29 03:54:20,953 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/362fbe53edaf44c8acf6a621729edb1f, entries=150, sequenceid=260, filesize=11.9 K 2023-03-29 03:54:20,953 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=100.63 KB/103050 for ee0c8ff38ef29b37b45871dfc89897f0 in 888ms, sequenceid=260, compaction requested=true 2023-03-29 03:54:20,953 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ee0c8ff38ef29b37b45871dfc89897f0: 2023-03-29 03:54:20,953 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. 2023-03-29 03:54:20,954 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:54:20,954 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:54:20,954 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:20,954 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:54:20,954 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:54:20,957 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:20,957 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:20,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:20,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:20,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:20,957 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:20,957 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:54:20,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:20,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:20,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:20,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:20,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:20,959 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:54:20,959 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@43120bf9[Count = 0] remaining members to acquire global barrier 2023-03-29 03:54:20,959 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:54:20,959 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:20,960 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:20,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:20,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:20,960 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:54:20,960 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:54:20,960 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:54:20,960 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:20,960 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:54:20,965 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:20,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on ee0c8ff38ef29b37b45871dfc89897f0 2023-03-29 03:54:20,965 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:54:20,965 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:20,965 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:54:20,965 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ee0c8ff38ef29b37b45871dfc89897f0 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-29 03:54:20,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:20,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:20,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:20,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:20,967 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680062060342, max timestamp=1680062060965Num uniques 859; 2023-03-29 03:54:20,967 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-29 03:54:20,967 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680062060342, max timestamp=1680062060965Num uniques 859; 2023-03-29 03:54:20,967 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-29 03:54:20,967 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680062060342, max timestamp=1680062060965Num uniques 859; 2023-03-29 03:54:20,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:20,967 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-29 03:54:20,968 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee0c8ff38ef29b37b45871dfc89897f0, store=A 2023-03-29 03:54:20,968 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-29 03:54:20,968 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee0c8ff38ef29b37b45871dfc89897f0, store=B 2023-03-29 03:54:20,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:20,968 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-29 03:54:20,968 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee0c8ff38ef29b37b45871dfc89897f0, store=C 2023-03-29 03:54:20,968 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-29 03:54:20,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:20,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:20,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:20,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:20,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:54:20,974 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:54:20,974 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:54:20,974 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:54:20,974 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:54:20,974 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:54:20,974 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:54:20,976 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:20,976 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:20,976 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:20,976 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:20,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:20,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:20,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:20,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:20,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:20,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:20,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:20,977 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:20,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:20,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:20,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:20,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:20,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:20,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:20,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:20,980 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:20,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:20,983 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:20,983 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:20,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:20,983 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:54:20,983 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:54:20,983 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:20,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:20,983 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:20,983 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:20,983 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:20,983 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:20,983 DEBUG [Thread-985] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:54:20,983 DEBUG [Thread-985] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:54:20,983 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:20,983 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:20,983 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:20,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:20,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:21,020 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=409, cellSize=18725, totalHeapSize=42776, min timestamp=1680062060990, max timestamp=1680062061018Num uniques 859; 2023-03-29 03:54:21,020 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=18725, suffix heap size=117064,new segment heap size=42776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=409 2023-03-29 03:54:21,020 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=409, cellSize=18725, totalHeapSize=42776, min timestamp=1680062060990, max timestamp=1680062061018Num uniques 859; 2023-03-29 03:54:21,021 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=18725, suffix heap size=117064,new segment heap size=42776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=409 2023-03-29 03:54:21,021 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=409, cellSize=18725, totalHeapSize=42776, min timestamp=1680062060990, max timestamp=1680062061018Num uniques 859; 2023-03-29 03:54:21,021 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=18725, suffix heap size=117064,new segment heap size=42776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=409 2023-03-29 03:54:21,048 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:21,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40120 deadline: 1680062121046, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:21,051 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:21,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40148 deadline: 1680062121047, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:21,052 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:21,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40142 deadline: 1680062121047, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:21,083 DEBUG [Thread-985] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:54:21,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:54:21,085 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:54:21,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:54:21,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:54:21,087 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:54:21,087 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:21,087 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:54:21,087 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:54:21,088 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:21,088 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:21,091 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:21,091 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:21,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:21,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:21,091 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:21,091 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:54:21,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:21,092 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:21,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:54:21,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:21,092 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:54:21,092 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:54:21,092 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:54:21,092 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:54:21,092 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:21,093 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:54:21,093 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:54:21,093 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:54:21,093 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. 2023-03-29 03:54:21,093 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. started... 2023-03-29 03:54:21,093 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. as already flushing 2023-03-29 03:54:21,093 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. 2023-03-29 03:54:21,093 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:54:21,093 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:54:21,093 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:21,093 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:54:21,093 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:54:21,095 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:21,095 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:21,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:21,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:21,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:21,095 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:21,095 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:54:21,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:21,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:21,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:21,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:21,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:21,096 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:54:21,096 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@18d3b7a5[Count = 0] remaining members to acquire global barrier 2023-03-29 03:54:21,096 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:54:21,097 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:21,098 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:21,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:21,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:21,098 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:54:21,098 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:54:21,098 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:54:21,098 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:21,098 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:54:21,099 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:21,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:21,099 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:54:21,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:21,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:21,100 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:21,100 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:54:21,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:21,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:21,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:21,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:21,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:21,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:21,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:21,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:54:21,102 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:54:21,102 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:54:21,102 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:54:21,102 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:54:21,102 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:54:21,102 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:54:21,104 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:21,104 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:21,104 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:21,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:21,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:21,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:21,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:21,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:21,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:21,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:21,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:21,105 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:21,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:21,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:21,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:21,106 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:21,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:21,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:21,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:21,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:21,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:21,108 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:21,108 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:21,108 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:21,108 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:21,108 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:21,108 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:21,108 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:21,108 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:21,108 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:21,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:21,108 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:21,108 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:54:21,109 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:54:21,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:21,109 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:54:21,109 DEBUG [Thread-985] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:54:21,109 DEBUG [Thread-985] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:54:21,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:21,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:21,149 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:21,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40120 deadline: 1680062121149, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:21,156 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:21,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40148 deadline: 1680062121152, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:21,156 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:21,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40142 deadline: 1680062121153, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:21,209 DEBUG [Thread-985] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:54:21,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:54:21,211 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:54:21,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:54:21,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:54:21,212 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:54:21,212 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:21,212 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:54:21,212 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:54:21,213 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:21,213 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:21,214 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:21,214 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:21,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:21,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:21,215 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:21,215 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:54:21,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:21,215 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:21,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:54:21,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:21,215 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:54:21,216 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:54:21,216 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:54:21,216 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:54:21,216 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:21,216 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:54:21,216 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:54:21,216 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:54:21,216 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. 2023-03-29 03:54:21,217 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. started... 2023-03-29 03:54:21,217 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. as already flushing 2023-03-29 03:54:21,217 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. 2023-03-29 03:54:21,217 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:54:21,217 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:54:21,217 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:21,217 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:54:21,217 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:54:21,218 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:21,218 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:21,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:21,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:21,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:21,218 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:21,218 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:54:21,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:21,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:21,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:21,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:21,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:21,220 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:54:21,220 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@756c8140[Count = 0] remaining members to acquire global barrier 2023-03-29 03:54:21,220 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:54:21,220 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:21,222 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:21,222 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:54:21,223 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:21,223 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:21,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:21,223 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:54:21,223 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:54:21,223 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:54:21,225 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:21,225 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:54:21,225 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:21,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:21,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:21,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:21,225 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:54:21,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:21,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:21,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:21,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:21,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:21,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:21,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:21,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:54:21,227 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:54:21,227 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:54:21,228 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:54:21,228 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:54:21,228 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:54:21,228 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:54:21,230 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:21,230 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:21,230 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:21,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:21,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:21,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:21,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:21,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:21,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:21,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:21,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:21,231 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:21,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:21,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:21,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:21,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:21,232 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:21,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:21,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:21,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:21,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:21,234 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:21,234 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:21,234 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:21,234 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:21,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:21,234 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:54:21,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:21,234 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:54:21,234 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:54:21,234 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:21,234 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:21,234 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:21,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:21,234 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:21,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:21,234 DEBUG [Thread-985] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:54:21,234 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:21,234 DEBUG [Thread-985] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:54:21,234 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:21,335 DEBUG [Thread-985] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:54:21,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:54:21,340 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:54:21,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:54:21,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:54:21,341 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:54:21,342 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:21,342 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:54:21,342 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:54:21,342 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:21,342 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:21,343 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:21,343 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:21,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:21,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:21,344 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:21,344 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:54:21,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:21,344 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:21,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:54:21,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:21,344 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:54:21,345 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:54:21,345 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:54:21,345 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:54:21,345 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:21,345 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:54:21,345 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:54:21,345 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:54:21,345 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. 2023-03-29 03:54:21,346 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. started... 2023-03-29 03:54:21,346 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. as already flushing 2023-03-29 03:54:21,346 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. 2023-03-29 03:54:21,346 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:54:21,346 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:54:21,346 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:21,346 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:54:21,346 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:54:21,347 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:21,347 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:21,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:21,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:21,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:21,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:21,348 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:21,348 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:54:21,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:21,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:21,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:21,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:21,349 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:54:21,349 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ddd4aba[Count = 0] remaining members to acquire global barrier 2023-03-29 03:54:21,349 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:54:21,349 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:21,351 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:21,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40120 deadline: 1680062121350, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:21,354 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:21,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:21,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:21,354 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:54:21,355 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:21,355 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:54:21,355 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:54:21,355 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:54:21,356 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:21,356 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:54:21,356 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:21,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:21,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:21,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:21,356 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:54:21,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:21,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:21,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:21,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:21,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:21,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:21,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:21,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:54:21,359 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:54:21,359 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:54:21,359 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:54:21,359 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:54:21,359 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:54:21,359 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:54:21,359 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:21,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40148 deadline: 1680062121357, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:21,360 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:21,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40142 deadline: 1680062121357, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:21,360 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:21,360 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:21,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:21,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:21,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:21,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:21,360 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:21,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:21,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:21,361 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:21,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:21,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:21,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:21,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:21,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:21,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:21,362 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:21,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:21,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:21,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:21,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:21,367 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:21,367 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:21,368 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:21,368 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:21,367 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:21,368 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:21,368 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:21,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:21,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:21,368 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:21,368 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:21,368 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:54:21,368 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:54:21,368 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:54:21,368 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:21,368 DEBUG [Thread-985] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:54:21,368 DEBUG [Thread-985] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:54:21,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:21,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:21,399 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303297b553edd067e4435bee15980d84e7c2b to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303297b553edd067e4435bee15980d84e7c2b 2023-03-29 03:54:21,404 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=281, memsize=13.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/A/5d91b581dce24de0a3dda2eb48ebc725 2023-03-29 03:54:21,422 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=281 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/B/f225a9c95c3d4fc48bb6cca927769502 2023-03-29 03:54:21,434 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=281 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/C/c5c070e21e6c4a108512e2e4dc30a536 2023-03-29 03:54:21,442 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/A/5d91b581dce24de0a3dda2eb48ebc725 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/5d91b581dce24de0a3dda2eb48ebc725 2023-03-29 03:54:21,449 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/5d91b581dce24de0a3dda2eb48ebc725, entries=200, sequenceid=281, filesize=32.3 K 2023-03-29 03:54:21,450 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/B/f225a9c95c3d4fc48bb6cca927769502 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/f225a9c95c3d4fc48bb6cca927769502 2023-03-29 03:54:21,455 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/f225a9c95c3d4fc48bb6cca927769502, entries=150, sequenceid=281, filesize=11.9 K 2023-03-29 03:54:21,455 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/C/c5c070e21e6c4a108512e2e4dc30a536 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/c5c070e21e6c4a108512e2e4dc30a536 2023-03-29 03:54:21,462 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/c5c070e21e6c4a108512e2e4dc30a536, entries=150, sequenceid=281, filesize=11.9 K 2023-03-29 03:54:21,463 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~35.78 KB/36640, heapSize ~86.50 KB/88576, currentSize=161.02 KB/164880 for ee0c8ff38ef29b37b45871dfc89897f0 in 498ms, sequenceid=281, compaction requested=true 2023-03-29 03:54:21,463 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ee0c8ff38ef29b37b45871dfc89897f0: 2023-03-29 03:54:21,463 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-03-29 03:54:21,464 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:54:21,464 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:54:21,464 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-29 03:54:21,464 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 8 store files, 0 compacting, 8 eligible, 16 blocking 2023-03-29 03:54:21,469 DEBUG [Thread-985] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:54:21,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:54:21,472 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:54:21,479 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 188904 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-03-29 03:54:21,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:54:21,479 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1912): ee0c8ff38ef29b37b45871dfc89897f0/A is initiating minor compaction (all files) 2023-03-29 03:54:21,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:54:21,479 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:54:21,479 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:21,479 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:54:21,479 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:54:21,479 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ee0c8ff38ef29b37b45871dfc89897f0/A in TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. 2023-03-29 03:54:21,479 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/fda0e2948cf147e9a1efaa7733436dad, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/3f0ed081b874469e8224777187f9d70a, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/0d5a970b176e497dbf41a2abbce4c4d4, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/7ecb6ce9ba7941ba80c3bed0abe1079d, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/3110413ad6634b3e8578f2e900123436, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/85008277797347258066bb667b5c6765, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/5d91b581dce24de0a3dda2eb48ebc725] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp, totalSize=184.5 K 2023-03-29 03:54:21,480 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:21,480 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:21,480 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting fda0e2948cf147e9a1efaa7733436dad, keycount=150, bloomtype=ROW, size=25.6 K, encoding=NONE, compression=NONE, seqNum=176, earliestPutTs=1680062054361 2023-03-29 03:54:21,481 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 3f0ed081b874469e8224777187f9d70a, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=194, earliestPutTs=1680062054483 2023-03-29 03:54:21,481 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 0d5a970b176e497dbf41a2abbce4c4d4, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=213, earliestPutTs=1680062055617 2023-03-29 03:54:21,482 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:21,482 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 8 files of size 96812 starting at candidate #0 after considering 21 permutations with 21 in ratio 2023-03-29 03:54:21,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:21,482 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1912): ee0c8ff38ef29b37b45871dfc89897f0/C is initiating minor compaction (all files) 2023-03-29 03:54:21,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:21,482 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ee0c8ff38ef29b37b45871dfc89897f0/C in TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. 2023-03-29 03:54:21,482 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 7ecb6ce9ba7941ba80c3bed0abe1079d, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=228, earliestPutTs=1680062057749 2023-03-29 03:54:21,482 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/0cd9cf0676354fc785b2c47b7596767f, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/176d6b10be3540cfa42110576e255a3d, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/445f38b2f5ed44feb05ef4041381382f, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/49676f4e28d94dd69ac114f618221c21, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/5a497fa220894f30bf9513d7734546c7, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/26be1f3657ce4301bc411a7166777796, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/362fbe53edaf44c8acf6a621729edb1f, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/c5c070e21e6c4a108512e2e4dc30a536] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp, totalSize=94.5 K 2023-03-29 03:54:21,482 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:21,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:21,482 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 0cd9cf0676354fc785b2c47b7596767f, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=159, earliestPutTs=1680062053732 2023-03-29 03:54:21,482 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 3110413ad6634b3e8578f2e900123436, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=245, earliestPutTs=1680062058069 2023-03-29 03:54:21,483 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 176d6b10be3540cfa42110576e255a3d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=176, earliestPutTs=1680062054361 2023-03-29 03:54:21,483 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 85008277797347258066bb667b5c6765, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=260, earliestPutTs=1680062059195 2023-03-29 03:54:21,484 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 5d91b581dce24de0a3dda2eb48ebc725, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=281, earliestPutTs=1680062060342 2023-03-29 03:54:21,484 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 445f38b2f5ed44feb05ef4041381382f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=194, earliestPutTs=1680062054483 2023-03-29 03:54:21,485 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 49676f4e28d94dd69ac114f618221c21, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=213, earliestPutTs=1680062055617 2023-03-29 03:54:21,486 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 5a497fa220894f30bf9513d7734546c7, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=228, earliestPutTs=1680062057749 2023-03-29 03:54:21,487 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 26be1f3657ce4301bc411a7166777796, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=245, earliestPutTs=1680062058069 2023-03-29 03:54:21,488 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 362fbe53edaf44c8acf6a621729edb1f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=260, earliestPutTs=1680062059195 2023-03-29 03:54:21,488 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:21,488 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:54:21,488 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:21,488 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting c5c070e21e6c4a108512e2e4dc30a536, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=281, earliestPutTs=1680062060342 2023-03-29 03:54:21,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:54:21,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:21,489 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:54:21,496 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:54:21,496 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:54:21,496 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:54:21,496 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:21,496 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:54:21,506 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:54:21,506 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:54:21,508 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. 2023-03-29 03:54:21,508 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. started... 2023-03-29 03:54:21,508 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ee0c8ff38ef29b37b45871dfc89897f0 3/3 column families, dataSize=161.02 KB heapSize=441.23 KB 2023-03-29 03:54:21,509 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee0c8ff38ef29b37b45871dfc89897f0, store=A 2023-03-29 03:54:21,509 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-29 03:54:21,509 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee0c8ff38ef29b37b45871dfc89897f0, store=B 2023-03-29 03:54:21,509 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-29 03:54:21,509 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee0c8ff38ef29b37b45871dfc89897f0, store=C 2023-03-29 03:54:21,510 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-29 03:54:21,516 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] throttle.PressureAwareThroughputController(145): ee0c8ff38ef29b37b45871dfc89897f0#C#compaction#176 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:54:21,539 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] throttle.PressureAwareThroughputController(145): ee0c8ff38ef29b37b45871dfc89897f0#A#compaction#175 average throughput is 0.66 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:54:21,569 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303294e2b42a3127a4425b8b5bf7331b68b77 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303294e2b42a3127a4425b8b5bf7331b68b77 2023-03-29 03:54:21,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on ee0c8ff38ef29b37b45871dfc89897f0 2023-03-29 03:54:21,655 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. as already flushing 2023-03-29 03:54:21,668 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:21,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40120 deadline: 1680062121665, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:21,672 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:21,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40148 deadline: 1680062121666, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:21,672 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:21,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40142 deadline: 1680062121666, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:21,771 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:21,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40120 deadline: 1680062121770, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:21,778 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:21,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40142 deadline: 1680062121773, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:21,779 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:21,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40148 deadline: 1680062121773, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:21,949 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/C/679d6f71cb274c37b93dcd22df4480ad as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/679d6f71cb274c37b93dcd22df4480ad 2023-03-29 03:54:21,955 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1652): Completed compaction of 8 (all) file(s) in ee0c8ff38ef29b37b45871dfc89897f0/C of ee0c8ff38ef29b37b45871dfc89897f0 into 679d6f71cb274c37b93dcd22df4480ad(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:54:21,955 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ee0c8ff38ef29b37b45871dfc89897f0: 2023-03-29 03:54:21,955 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0., storeName=ee0c8ff38ef29b37b45871dfc89897f0/C, priority=8, startTime=1680062061464; duration=0sec 2023-03-29 03:54:21,955 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:54:21,955 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-03-29 03:54:21,959 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 84844 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-03-29 03:54:21,959 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1912): ee0c8ff38ef29b37b45871dfc89897f0/B is initiating minor compaction (all files) 2023-03-29 03:54:21,959 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ee0c8ff38ef29b37b45871dfc89897f0/B in TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. 2023-03-29 03:54:21,959 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/e61e6c7532674f6298e462f6f02afc5c, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/6e49a08930004a6096945b85cb0c9b78, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/e7627510cac2403ab1578315b8034ee5, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/eb60a436a7fc4f439a5c78a68210c89f, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/20a663170f70472599a8d4c8e072398a, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/0bd911f13f094a62bbedd35e6faf71a0, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/f225a9c95c3d4fc48bb6cca927769502] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp, totalSize=82.9 K 2023-03-29 03:54:21,959 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting e61e6c7532674f6298e462f6f02afc5c, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=176, earliestPutTs=1680062054361 2023-03-29 03:54:21,960 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 6e49a08930004a6096945b85cb0c9b78, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=194, earliestPutTs=1680062054483 2023-03-29 03:54:21,962 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting e7627510cac2403ab1578315b8034ee5, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=213, earliestPutTs=1680062055617 2023-03-29 03:54:21,962 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting eb60a436a7fc4f439a5c78a68210c89f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=228, earliestPutTs=1680062057749 2023-03-29 03:54:21,969 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 20a663170f70472599a8d4c8e072398a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=245, earliestPutTs=1680062058069 2023-03-29 03:54:21,970 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 0bd911f13f094a62bbedd35e6faf71a0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=260, earliestPutTs=1680062059195 2023-03-29 03:54:21,972 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting f225a9c95c3d4fc48bb6cca927769502, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=281, earliestPutTs=1680062060342 2023-03-29 03:54:21,975 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:21,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40120 deadline: 1680062121972, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:21,981 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:21,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40142 deadline: 1680062121980, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:21,982 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:21,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40148 deadline: 1680062121981, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:21,984 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=317, memsize=53.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/A/84d423adfc6d4ffb98ddc2e54747c29d 2023-03-29 03:54:22,010 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] throttle.PressureAwareThroughputController(145): ee0c8ff38ef29b37b45871dfc89897f0#B#compaction#178 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:54:22,053 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/B/9ec3cf0242d448aabca6081ed30f18ee as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/9ec3cf0242d448aabca6081ed30f18ee 2023-03-29 03:54:22,061 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in ee0c8ff38ef29b37b45871dfc89897f0/B of ee0c8ff38ef29b37b45871dfc89897f0 into 9ec3cf0242d448aabca6081ed30f18ee(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:54:22,061 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ee0c8ff38ef29b37b45871dfc89897f0: 2023-03-29 03:54:22,061 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0., storeName=ee0c8ff38ef29b37b45871dfc89897f0/B, priority=9, startTime=1680062061464; duration=0sec 2023-03-29 03:54:22,061 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:54:22,278 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:22,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40120 deadline: 1680062122277, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:22,286 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:22,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40148 deadline: 1680062122284, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:22,286 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:22,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40142 deadline: 1680062122284, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:22,438 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/A/f4c67b9d9ba04dc7bf95c7e502fbf7b8 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/f4c67b9d9ba04dc7bf95c7e502fbf7b8 2023-03-29 03:54:22,444 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=317 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/B/42ebf4470f2d4934a91ed79736a69e64 2023-03-29 03:54:22,450 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in ee0c8ff38ef29b37b45871dfc89897f0/A of ee0c8ff38ef29b37b45871dfc89897f0 into f4c67b9d9ba04dc7bf95c7e502fbf7b8(size=26.0 K), total size for store is 26.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:54:22,450 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ee0c8ff38ef29b37b45871dfc89897f0: 2023-03-29 03:54:22,450 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0., storeName=ee0c8ff38ef29b37b45871dfc89897f0/A, priority=9, startTime=1680062061463; duration=0sec 2023-03-29 03:54:22,450 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:54:22,463 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=317 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/C/95627fb6b16f43d6af9ba010989d502c 2023-03-29 03:54:22,468 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/A/84d423adfc6d4ffb98ddc2e54747c29d as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/84d423adfc6d4ffb98ddc2e54747c29d 2023-03-29 03:54:22,472 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/84d423adfc6d4ffb98ddc2e54747c29d, entries=150, sequenceid=317, filesize=25.4 K 2023-03-29 03:54:22,473 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/B/42ebf4470f2d4934a91ed79736a69e64 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/42ebf4470f2d4934a91ed79736a69e64 2023-03-29 03:54:22,478 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/42ebf4470f2d4934a91ed79736a69e64, entries=150, sequenceid=317, filesize=11.9 K 2023-03-29 03:54:22,481 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/C/95627fb6b16f43d6af9ba010989d502c as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/95627fb6b16f43d6af9ba010989d502c 2023-03-29 03:54:22,491 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/95627fb6b16f43d6af9ba010989d502c, entries=150, sequenceid=317, filesize=11.9 K 2023-03-29 03:54:22,491 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~161.02 KB/164880, heapSize ~441.19 KB/451776, currentSize=33.54 KB/34350 for ee0c8ff38ef29b37b45871dfc89897f0 in 983ms, sequenceid=317, compaction requested=false 2023-03-29 03:54:22,492 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ee0c8ff38ef29b37b45871dfc89897f0: 2023-03-29 03:54:22,492 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. 2023-03-29 03:54:22,492 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:54:22,492 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:54:22,492 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:22,492 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:54:22,492 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:54:22,494 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:22,494 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:22,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:22,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:22,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:22,494 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:22,494 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:54:22,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:22,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:22,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:22,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:22,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:22,508 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:54:22,508 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3a5fdd6d[Count = 0] remaining members to acquire global barrier 2023-03-29 03:54:22,508 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:54:22,508 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:22,510 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:22,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:22,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:22,510 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:54:22,510 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:54:22,510 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:54:22,511 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:22,511 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:54:22,513 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:54:22,513 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:22,513 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:54:22,513 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:22,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:22,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:22,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:22,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:22,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:22,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:22,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:22,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:22,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:22,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:22,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:54:22,516 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:54:22,516 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:54:22,516 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:54:22,516 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:54:22,516 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:54:22,516 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:54:22,519 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:22,519 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:22,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:22,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:22,519 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:22,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:22,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:22,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:22,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:22,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:22,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:22,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:22,521 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:22,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:22,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:22,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:22,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:22,523 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:22,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:22,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:22,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:22,529 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:22,529 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:22,529 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:54:22,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:22,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:22,529 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:22,529 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:54:22,530 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:54:22,529 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:22,529 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:22,530 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:22,530 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:22,530 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:22,530 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:22,530 DEBUG [Thread-985] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:54:22,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:22,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:22,530 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:22,530 DEBUG [Thread-985] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:54:22,630 DEBUG [Thread-985] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:54:22,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:54:22,633 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:54:22,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:54:22,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:54:22,649 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:54:22,649 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:22,649 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:54:22,649 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:54:22,650 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:22,650 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:22,652 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:22,652 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:22,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:22,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:22,652 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:22,652 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:54:22,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:22,653 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:22,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:54:22,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:22,653 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:54:22,656 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:54:22,656 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:54:22,656 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:54:22,656 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:22,656 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:54:22,656 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:54:22,656 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:54:22,656 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. 2023-03-29 03:54:22,656 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. started... 2023-03-29 03:54:22,657 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ee0c8ff38ef29b37b45871dfc89897f0 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-03-29 03:54:22,657 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee0c8ff38ef29b37b45871dfc89897f0, store=A 2023-03-29 03:54:22,657 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:54:22,657 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee0c8ff38ef29b37b45871dfc89897f0, store=B 2023-03-29 03:54:22,657 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:54:22,657 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee0c8ff38ef29b37b45871dfc89897f0, store=C 2023-03-29 03:54:22,657 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:54:22,696 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230329d6f3a9ea320b4743b498521ff3f428e3 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230329d6f3a9ea320b4743b498521ff3f428e3 2023-03-29 03:54:22,751 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=328, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/A/48a0bdf086464a46b94c6707ecb26a38 2023-03-29 03:54:22,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on ee0c8ff38ef29b37b45871dfc89897f0 2023-03-29 03:54:22,792 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. as already flushing 2023-03-29 03:54:22,826 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1680062062812, max timestamp=1680062062823Num uniques 859; 2023-03-29 03:54:22,826 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1680062062812, max timestamp=1680062062823Num uniques 859; 2023-03-29 03:54:22,826 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-03-29 03:54:22,826 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1680062062812, max timestamp=1680062062823Num uniques 859; 2023-03-29 03:54:22,826 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-03-29 03:54:22,828 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-03-29 03:54:22,854 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:22,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40142 deadline: 1680062122849, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:22,855 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:22,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40120 deadline: 1680062122849, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:22,856 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:22,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40148 deadline: 1680062122852, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:22,957 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:22,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40142 deadline: 1680062122956, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:22,958 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:22,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40120 deadline: 1680062122956, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:22,960 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:22,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40148 deadline: 1680062122959, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:23,159 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:23,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40120 deadline: 1680062123159, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:23,161 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:23,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40142 deadline: 1680062123160, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:23,162 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:23,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40148 deadline: 1680062123161, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:23,201 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=328 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/B/598c8fa333a14f6487ed9d8cdcf695ff 2023-03-29 03:54:23,215 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=328 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/C/fc31bd2e87874274a4ae87be71c1fda6 2023-03-29 03:54:23,220 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/A/48a0bdf086464a46b94c6707ecb26a38 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/48a0bdf086464a46b94c6707ecb26a38 2023-03-29 03:54:23,225 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/48a0bdf086464a46b94c6707ecb26a38, entries=150, sequenceid=328, filesize=25.4 K 2023-03-29 03:54:23,226 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/B/598c8fa333a14f6487ed9d8cdcf695ff as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/598c8fa333a14f6487ed9d8cdcf695ff 2023-03-29 03:54:23,233 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/598c8fa333a14f6487ed9d8cdcf695ff, entries=150, sequenceid=328, filesize=11.9 K 2023-03-29 03:54:23,235 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/C/fc31bd2e87874274a4ae87be71c1fda6 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/fc31bd2e87874274a4ae87be71c1fda6 2023-03-29 03:54:23,251 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/fc31bd2e87874274a4ae87be71c1fda6, entries=150, sequenceid=328, filesize=11.9 K 2023-03-29 03:54:23,252 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=154.31 KB/158010 for ee0c8ff38ef29b37b45871dfc89897f0 in 595ms, sequenceid=328, compaction requested=true 2023-03-29 03:54:23,253 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ee0c8ff38ef29b37b45871dfc89897f0: 2023-03-29 03:54:23,253 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. 2023-03-29 03:54:23,253 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:54:23,253 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:54:23,253 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:23,253 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:54:23,253 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:54:23,255 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:23,255 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:23,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:23,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:23,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:23,255 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:23,255 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:54:23,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:23,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:23,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:23,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:23,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:23,257 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:54:23,257 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d40d806[Count = 0] remaining members to acquire global barrier 2023-03-29 03:54:23,257 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:54:23,257 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:23,258 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:23,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:23,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:23,258 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:54:23,259 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:54:23,259 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:54:23,259 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:23,259 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:54:23,260 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:23,260 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:54:23,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:23,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:23,260 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:23,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:23,260 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:54:23,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:23,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:23,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:23,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:23,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:23,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:23,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:23,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:54:23,263 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:54:23,263 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:54:23,263 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:54:23,263 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:54:23,263 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:54:23,263 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:54:23,265 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:23,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:23,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:23,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:23,265 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:23,265 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:23,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:23,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:23,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:23,266 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:23,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:23,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:23,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:23,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:23,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:23,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:23,267 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:23,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:23,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:23,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:23,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:23,269 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:23,269 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:23,269 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:23,269 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:23,269 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:54:23,269 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:54:23,269 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:23,269 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:23,269 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:23,269 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:23,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:23,270 DEBUG [Thread-985] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:54:23,269 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:23,269 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:23,270 DEBUG [Thread-985] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:54:23,270 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:54:23,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:23,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:23,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:23,370 DEBUG [Thread-985] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:54:23,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:54:23,372 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:54:23,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:54:23,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:54:23,373 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:54:23,373 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:23,373 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:54:23,373 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:54:23,374 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:23,374 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:23,375 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:23,375 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:23,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:23,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:23,376 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:23,376 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:54:23,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:23,376 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:23,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:54:23,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:23,377 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:54:23,377 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:54:23,377 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:54:23,377 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:54:23,377 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:23,377 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:54:23,377 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:54:23,377 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:54:23,377 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. 2023-03-29 03:54:23,377 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. started... 2023-03-29 03:54:23,378 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ee0c8ff38ef29b37b45871dfc89897f0 3/3 column families, dataSize=154.31 KB heapSize=426 KB 2023-03-29 03:54:23,378 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee0c8ff38ef29b37b45871dfc89897f0, store=A 2023-03-29 03:54:23,378 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-29 03:54:23,378 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee0c8ff38ef29b37b45871dfc89897f0, store=B 2023-03-29 03:54:23,378 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-29 03:54:23,378 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee0c8ff38ef29b37b45871dfc89897f0, store=C 2023-03-29 03:54:23,378 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-29 03:54:23,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on ee0c8ff38ef29b37b45871dfc89897f0 2023-03-29 03:54:23,464 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. as already flushing 2023-03-29 03:54:23,472 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:23,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40120 deadline: 1680062123468, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:23,472 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:23,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40142 deadline: 1680062123468, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:23,473 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:23,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40148 deadline: 1680062123469, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:23,575 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:23,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40120 deadline: 1680062123573, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:23,576 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:23,576 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:23,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40142 deadline: 1680062123573, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:23,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40148 deadline: 1680062123574, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:23,779 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:23,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40120 deadline: 1680062123777, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:23,779 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:23,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40148 deadline: 1680062123777, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:23,779 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:23,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40142 deadline: 1680062123777, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:23,792 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303290951514e9bff4e8b883f3177143217c3 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303290951514e9bff4e8b883f3177143217c3 2023-03-29 03:54:24,082 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:24,082 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:24,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40142 deadline: 1680062124080, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:24,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40148 deadline: 1680062124080, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:24,082 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:24,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40120 deadline: 1680062124082, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:24,201 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=364, memsize=51.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/A/7b25a644464b4ea8b5d49653d4f65ed8 2023-03-29 03:54:24,218 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=364 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/B/10c77866d5bb48428ea73938624f67cf 2023-03-29 03:54:24,585 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:24,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40120 deadline: 1680062124585, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:24,588 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:24,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40142 deadline: 1680062124585, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:24,588 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:24,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40148 deadline: 1680062124585, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:24,633 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=364 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/C/e0ed94b94124416ea2405f2b4321b442 2023-03-29 03:54:24,637 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/A/7b25a644464b4ea8b5d49653d4f65ed8 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/7b25a644464b4ea8b5d49653d4f65ed8 2023-03-29 03:54:24,642 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/7b25a644464b4ea8b5d49653d4f65ed8, entries=150, sequenceid=364, filesize=25.4 K 2023-03-29 03:54:24,642 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/B/10c77866d5bb48428ea73938624f67cf as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/10c77866d5bb48428ea73938624f67cf 2023-03-29 03:54:24,646 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/10c77866d5bb48428ea73938624f67cf, entries=150, sequenceid=364, filesize=11.9 K 2023-03-29 03:54:24,647 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/C/e0ed94b94124416ea2405f2b4321b442 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/e0ed94b94124416ea2405f2b4321b442 2023-03-29 03:54:24,658 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/e0ed94b94124416ea2405f2b4321b442, entries=150, sequenceid=364, filesize=11.9 K 2023-03-29 03:54:24,658 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~154.31 KB/158010, heapSize ~425.95 KB/436176, currentSize=33.54 KB/34350 for ee0c8ff38ef29b37b45871dfc89897f0 in 1281ms, sequenceid=364, compaction requested=true 2023-03-29 03:54:24,659 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ee0c8ff38ef29b37b45871dfc89897f0: 2023-03-29 03:54:24,659 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. 2023-03-29 03:54:24,659 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:54:24,659 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:54:24,659 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:24,659 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:54:24,659 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:54:24,661 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:24,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:24,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:24,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:24,661 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:24,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:24,661 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:24,661 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:54:24,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:24,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:24,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:24,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:24,663 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:54:24,663 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1dea5486[Count = 0] remaining members to acquire global barrier 2023-03-29 03:54:24,663 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:54:24,663 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:24,665 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:24,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:24,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:24,665 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:54:24,665 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:54:24,665 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:54:24,665 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:24,665 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:54:24,669 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:54:24,669 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:24,669 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:24,669 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:54:24,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:24,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:24,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:24,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:24,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:24,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:24,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:24,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:24,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:24,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:24,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:54:24,673 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:54:24,673 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:54:24,673 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:54:24,673 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:54:24,673 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:54:24,673 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:54:24,675 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:24,675 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:24,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:24,675 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:24,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:24,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:24,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:24,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:24,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:24,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:24,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:24,676 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:24,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:24,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:24,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:24,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:24,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:24,677 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:24,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:24,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:24,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:24,683 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:24,683 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:24,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:24,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:24,683 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:24,683 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:24,684 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:24,684 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:24,684 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:24,684 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:24,684 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:24,684 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:24,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:24,683 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:54:24,684 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:54:24,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:24,684 DEBUG [Thread-985] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:54:24,684 DEBUG [Thread-985] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:54:24,784 DEBUG [Thread-985] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:54:24,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:54:24,787 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:54:24,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:54:24,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:54:24,788 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:54:24,789 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:24,789 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:54:24,789 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:54:24,789 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:24,789 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:24,791 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:24,791 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:24,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:24,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:24,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:24,791 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:24,792 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:54:24,792 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:24,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:54:24,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:24,792 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:54:24,793 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:54:24,793 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:54:24,793 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:54:24,793 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:24,793 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:54:24,793 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:54:24,793 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:54:24,793 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. 2023-03-29 03:54:24,793 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. started... 2023-03-29 03:54:24,794 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ee0c8ff38ef29b37b45871dfc89897f0 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-03-29 03:54:24,794 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee0c8ff38ef29b37b45871dfc89897f0, store=A 2023-03-29 03:54:24,794 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:54:24,794 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee0c8ff38ef29b37b45871dfc89897f0, store=B 2023-03-29 03:54:24,794 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:54:24,794 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee0c8ff38ef29b37b45871dfc89897f0, store=C 2023-03-29 03:54:24,794 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:54:24,811 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303293160144868894cae9f5adc7b59c0f97a to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303293160144868894cae9f5adc7b59c0f97a 2023-03-29 03:54:25,225 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=372, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/A/d9cb6508c43d48328ac893ce7b94a47b 2023-03-29 03:54:25,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on ee0c8ff38ef29b37b45871dfc89897f0 2023-03-29 03:54:25,604 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. as already flushing 2023-03-29 03:54:25,647 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680062065634, max timestamp=1680062065645Num uniques 859; 2023-03-29 03:54:25,647 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-29 03:54:25,648 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680062065634, max timestamp=1680062065645Num uniques 859; 2023-03-29 03:54:25,648 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-29 03:54:25,648 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680062065634, max timestamp=1680062065645Num uniques 859; 2023-03-29 03:54:25,649 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-29 03:54:25,658 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=372 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/B/6bac5dcfc5c641f3a51c26d57357673c 2023-03-29 03:54:25,680 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1680062065662, max timestamp=1680062065677Num uniques 1018; 2023-03-29 03:54:25,680 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1680062065662, max timestamp=1680062065677Num uniques 1018; 2023-03-29 03:54:25,680 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-03-29 03:54:25,680 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-03-29 03:54:25,681 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1680062065662, max timestamp=1680062065677Num uniques 1018; 2023-03-29 03:54:25,681 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-03-29 03:54:25,701 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=372 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/C/ec55b793b5674e3ba492c683b3e3825e 2023-03-29 03:54:25,706 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/A/d9cb6508c43d48328ac893ce7b94a47b as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/d9cb6508c43d48328ac893ce7b94a47b 2023-03-29 03:54:25,712 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/d9cb6508c43d48328ac893ce7b94a47b, entries=100, sequenceid=372, filesize=18.6 K 2023-03-29 03:54:25,713 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/B/6bac5dcfc5c641f3a51c26d57357673c as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/6bac5dcfc5c641f3a51c26d57357673c 2023-03-29 03:54:25,718 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/6bac5dcfc5c641f3a51c26d57357673c, entries=100, sequenceid=372, filesize=9.5 K 2023-03-29 03:54:25,719 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/C/ec55b793b5674e3ba492c683b3e3825e as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/ec55b793b5674e3ba492c683b3e3825e 2023-03-29 03:54:25,725 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=377, cellSize=17262, totalHeapSize=39448, min timestamp=1680062065699, max timestamp=1680062065722Num uniques 1077; 2023-03-29 03:54:25,725 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=17262, suffix heap size=139976,new segment heap size=39448 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=377 2023-03-29 03:54:25,725 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=377, cellSize=17262, totalHeapSize=39448, min timestamp=1680062065699, max timestamp=1680062065722Num uniques 1077; 2023-03-29 03:54:25,725 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=17262, suffix heap size=139976,new segment heap size=39448 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=377 2023-03-29 03:54:25,726 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=377, cellSize=17262, totalHeapSize=39448, min timestamp=1680062065699, max timestamp=1680062065723Num uniques 1077; 2023-03-29 03:54:25,726 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=17262, suffix heap size=139976,new segment heap size=39448 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=377 2023-03-29 03:54:25,727 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/ec55b793b5674e3ba492c683b3e3825e, entries=100, sequenceid=372, filesize=9.5 K 2023-03-29 03:54:25,728 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=60.38 KB/61830 for ee0c8ff38ef29b37b45871dfc89897f0 in 935ms, sequenceid=372, compaction requested=true 2023-03-29 03:54:25,728 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ee0c8ff38ef29b37b45871dfc89897f0: 2023-03-29 03:54:25,728 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. 2023-03-29 03:54:25,728 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:54:25,728 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:54:25,728 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:25,728 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:54:25,728 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:54:25,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on ee0c8ff38ef29b37b45871dfc89897f0 2023-03-29 03:54:25,729 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ee0c8ff38ef29b37b45871dfc89897f0 3/3 column families, dataSize=67.09 KB heapSize=165.33 KB 2023-03-29 03:54:25,729 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee0c8ff38ef29b37b45871dfc89897f0, store=A 2023-03-29 03:54:25,729 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-29 03:54:25,729 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee0c8ff38ef29b37b45871dfc89897f0, store=B 2023-03-29 03:54:25,729 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-29 03:54:25,729 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee0c8ff38ef29b37b45871dfc89897f0, store=C 2023-03-29 03:54:25,729 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-29 03:54:25,730 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:25,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:25,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:25,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:25,731 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:25,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:25,731 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:25,731 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:54:25,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:25,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:25,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:25,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:25,733 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:54:25,733 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@77d6f615[Count = 0] remaining members to acquire global barrier 2023-03-29 03:54:25,733 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:54:25,733 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:25,735 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:25,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:25,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:25,735 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:54:25,735 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:54:25,735 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:54:25,737 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:25,737 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:54:25,739 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:25,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:25,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:25,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:25,740 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:54:25,740 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:25,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:25,740 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:54:25,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:25,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:25,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:25,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:25,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:25,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:25,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:54:25,743 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:54:25,743 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:54:25,743 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:54:25,743 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:54:25,743 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:54:25,743 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:54:25,745 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:25,745 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:25,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:25,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:25,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:25,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:25,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:25,745 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:25,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:25,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:25,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:25,747 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:25,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:25,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:25,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:25,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:25,749 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:25,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:25,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:25,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:25,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:25,752 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:25,752 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:25,753 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:25,753 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:25,753 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:25,753 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:25,753 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:25,753 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:25,753 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:25,753 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:25,753 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:54:25,753 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:54:25,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:25,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:25,753 DEBUG [Thread-985] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:54:25,753 DEBUG [Thread-985] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:54:25,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:25,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:25,754 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:54:25,770 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230329d197d0f0bf9143c78a392ba1efcef176 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230329d197d0f0bf9143c78a392ba1efcef176 2023-03-29 03:54:25,778 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1680062065753, max timestamp=1680062065776Num uniques 859; 2023-03-29 03:54:25,778 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1680062065753, max timestamp=1680062065776Num uniques 859; 2023-03-29 03:54:25,778 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-03-29 03:54:25,778 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1680062065753, max timestamp=1680062065776Num uniques 859; 2023-03-29 03:54:25,778 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-03-29 03:54:25,778 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-03-29 03:54:25,811 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:25,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40120 deadline: 1680062125804, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:25,811 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:25,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40142 deadline: 1680062125805, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:25,811 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:25,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40148 deadline: 1680062125805, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:25,816 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=429, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/A/095f42e40c1b4d29bd6ad059f691b7ba 2023-03-29 03:54:25,853 DEBUG [Thread-985] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:54:25,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:54:25,864 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:54:25,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:54:25,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:54:25,878 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:54:25,878 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=429 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/B/8ca887db35ab496d86c542caaf1381dd 2023-03-29 03:54:25,878 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:25,878 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:54:25,878 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:54:25,878 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:25,878 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:25,880 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:25,880 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:25,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:25,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:25,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:25,881 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:25,881 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:54:25,881 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:25,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:54:25,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:25,882 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:54:25,888 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:54:25,888 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:54:25,888 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:54:25,888 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:25,888 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:54:25,902 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:54:25,902 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:54:25,902 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. 2023-03-29 03:54:25,902 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. started... 2023-03-29 03:54:25,903 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. as already flushing 2023-03-29 03:54:25,903 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. 2023-03-29 03:54:25,904 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:54:25,904 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:54:25,904 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:25,904 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:54:25,904 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:54:25,907 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:25,907 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:25,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:25,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:25,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:25,908 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:25,908 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:54:25,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:25,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:25,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:25,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:25,914 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=429 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/C/18d80ceef9ac42d19f94a1e2460ce51a 2023-03-29 03:54:25,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:25,915 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:54:25,915 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3326f937[Count = 0] remaining members to acquire global barrier 2023-03-29 03:54:25,915 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:54:25,915 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:25,916 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:25,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40120 deadline: 1680062125912, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:25,916 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:25,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40142 deadline: 1680062125912, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:25,917 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:25,917 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:25,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40148 deadline: 1680062125913, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:25,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:25,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:25,917 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:54:25,917 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:54:25,917 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:25,918 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:54:25,918 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:54:25,919 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:25,919 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:54:25,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:25,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:25,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:25,919 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:25,920 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:54:25,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:25,920 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/A/095f42e40c1b4d29bd6ad059f691b7ba as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/095f42e40c1b4d29bd6ad059f691b7ba 2023-03-29 03:54:25,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:25,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:25,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:25,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:25,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:25,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:25,925 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/095f42e40c1b4d29bd6ad059f691b7ba, entries=200, sequenceid=429, filesize=32.3 K 2023-03-29 03:54:25,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:54:25,925 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:54:25,925 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:54:25,925 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:54:25,925 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:54:25,925 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:54:25,926 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:54:25,926 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/B/8ca887db35ab496d86c542caaf1381dd as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/8ca887db35ab496d86c542caaf1381dd 2023-03-29 03:54:25,928 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:25,928 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:25,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:25,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:25,928 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:25,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:25,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:25,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:25,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:25,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:25,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:25,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:25,929 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:25,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:25,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:25,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:25,931 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:25,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:25,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:25,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:25,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:25,935 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:25,935 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:25,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:25,935 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:25,935 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:25,935 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:54:25,936 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:54:25,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:25,935 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:54:25,935 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:25,936 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:25,936 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:25,936 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:25,936 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:25,936 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:25,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:25,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:25,936 DEBUG [Thread-985] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:54:25,936 DEBUG [Thread-985] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:54:25,951 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/8ca887db35ab496d86c542caaf1381dd, entries=150, sequenceid=429, filesize=11.9 K 2023-03-29 03:54:25,952 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/C/18d80ceef9ac42d19f94a1e2460ce51a as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/18d80ceef9ac42d19f94a1e2460ce51a 2023-03-29 03:54:25,957 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/18d80ceef9ac42d19f94a1e2460ce51a, entries=150, sequenceid=429, filesize=11.9 K 2023-03-29 03:54:25,958 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~165.28 KB/169248, currentSize=134.18 KB/137400 for ee0c8ff38ef29b37b45871dfc89897f0 in 230ms, sequenceid=429, compaction requested=true 2023-03-29 03:54:25,958 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ee0c8ff38ef29b37b45871dfc89897f0: 2023-03-29 03:54:25,958 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:54:25,958 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-29 03:54:25,958 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-29 03:54:25,958 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:54:25,958 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:54:25,960 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 156848 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-29 03:54:25,960 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1912): ee0c8ff38ef29b37b45871dfc89897f0/A is initiating minor compaction (all files) 2023-03-29 03:54:25,960 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 71252 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-29 03:54:25,960 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ee0c8ff38ef29b37b45871dfc89897f0/A in TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. 2023-03-29 03:54:25,961 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1912): ee0c8ff38ef29b37b45871dfc89897f0/B is initiating minor compaction (all files) 2023-03-29 03:54:25,961 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/f4c67b9d9ba04dc7bf95c7e502fbf7b8, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/84d423adfc6d4ffb98ddc2e54747c29d, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/48a0bdf086464a46b94c6707ecb26a38, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/7b25a644464b4ea8b5d49653d4f65ed8, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/d9cb6508c43d48328ac893ce7b94a47b, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/095f42e40c1b4d29bd6ad059f691b7ba] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp, totalSize=153.2 K 2023-03-29 03:54:25,961 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ee0c8ff38ef29b37b45871dfc89897f0/B in TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. 2023-03-29 03:54:25,961 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/9ec3cf0242d448aabca6081ed30f18ee, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/42ebf4470f2d4934a91ed79736a69e64, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/598c8fa333a14f6487ed9d8cdcf695ff, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/10c77866d5bb48428ea73938624f67cf, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/6bac5dcfc5c641f3a51c26d57357673c, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/8ca887db35ab496d86c542caaf1381dd] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp, totalSize=69.6 K 2023-03-29 03:54:25,961 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting f4c67b9d9ba04dc7bf95c7e502fbf7b8, keycount=150, bloomtype=ROW, size=26.0 K, encoding=NONE, compression=NONE, seqNum=281, earliestPutTs=1680062060342 2023-03-29 03:54:25,961 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 84d423adfc6d4ffb98ddc2e54747c29d, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=317, earliestPutTs=1680062061045 2023-03-29 03:54:25,962 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 48a0bdf086464a46b94c6707ecb26a38, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=328, earliestPutTs=1680062061654 2023-03-29 03:54:25,962 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 9ec3cf0242d448aabca6081ed30f18ee, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=281, earliestPutTs=1680062060342 2023-03-29 03:54:25,962 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 7b25a644464b4ea8b5d49653d4f65ed8, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=364, earliestPutTs=1680062062847 2023-03-29 03:54:25,963 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting d9cb6508c43d48328ac893ce7b94a47b, keycount=100, bloomtype=ROW, size=18.6 K, encoding=NONE, compression=NONE, seqNum=372, earliestPutTs=1680062063463 2023-03-29 03:54:25,963 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 095f42e40c1b4d29bd6ad059f691b7ba, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=429, earliestPutTs=1680062065713 2023-03-29 03:54:25,964 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 42ebf4470f2d4934a91ed79736a69e64, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=317, earliestPutTs=1680062061045 2023-03-29 03:54:25,966 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 598c8fa333a14f6487ed9d8cdcf695ff, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=328, earliestPutTs=1680062061654 2023-03-29 03:54:25,968 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 10c77866d5bb48428ea73938624f67cf, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=364, earliestPutTs=1680062062847 2023-03-29 03:54:25,972 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 6bac5dcfc5c641f3a51c26d57357673c, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=372, earliestPutTs=1680062063463 2023-03-29 03:54:25,973 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 8ca887db35ab496d86c542caaf1381dd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=429, earliestPutTs=1680062065722 2023-03-29 03:54:25,999 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] throttle.PressureAwareThroughputController(145): ee0c8ff38ef29b37b45871dfc89897f0#B#compaction#193 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:54:26,023 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] throttle.PressureAwareThroughputController(145): ee0c8ff38ef29b37b45871dfc89897f0#A#compaction#194 average throughput is 0.86 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:54:26,035 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/B/d85df311c0f349299d2007f6b7007c13 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/d85df311c0f349299d2007f6b7007c13 2023-03-29 03:54:26,036 DEBUG [Thread-985] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:54:26,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:54:26,040 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:54:26,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:54:26,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:54:26,053 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:54:26,053 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:26,053 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:54:26,053 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:54:26,054 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:26,054 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:26,055 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:26,055 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:26,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:26,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:26,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:26,057 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:26,057 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in ee0c8ff38ef29b37b45871dfc89897f0/B of ee0c8ff38ef29b37b45871dfc89897f0 into d85df311c0f349299d2007f6b7007c13(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:54:26,057 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ee0c8ff38ef29b37b45871dfc89897f0: 2023-03-29 03:54:26,057 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:26,057 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:54:26,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:54:26,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:26,057 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:54:26,057 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0., storeName=ee0c8ff38ef29b37b45871dfc89897f0/B, priority=10, startTime=1680062065958; duration=0sec 2023-03-29 03:54:26,058 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:54:26,058 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-29 03:54:26,062 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 71218 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-29 03:54:26,062 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1912): ee0c8ff38ef29b37b45871dfc89897f0/C is initiating minor compaction (all files) 2023-03-29 03:54:26,062 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ee0c8ff38ef29b37b45871dfc89897f0/C in TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. 2023-03-29 03:54:26,062 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/679d6f71cb274c37b93dcd22df4480ad, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/95627fb6b16f43d6af9ba010989d502c, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/fc31bd2e87874274a4ae87be71c1fda6, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/e0ed94b94124416ea2405f2b4321b442, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/ec55b793b5674e3ba492c683b3e3825e, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/18d80ceef9ac42d19f94a1e2460ce51a] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp, totalSize=69.5 K 2023-03-29 03:54:26,063 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 679d6f71cb274c37b93dcd22df4480ad, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=281, earliestPutTs=1680062060342 2023-03-29 03:54:26,063 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 95627fb6b16f43d6af9ba010989d502c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=317, earliestPutTs=1680062061045 2023-03-29 03:54:26,064 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting fc31bd2e87874274a4ae87be71c1fda6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=328, earliestPutTs=1680062061654 2023-03-29 03:54:26,065 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting e0ed94b94124416ea2405f2b4321b442, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=364, earliestPutTs=1680062062847 2023-03-29 03:54:26,065 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting ec55b793b5674e3ba492c683b3e3825e, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=372, earliestPutTs=1680062063463 2023-03-29 03:54:26,066 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 18d80ceef9ac42d19f94a1e2460ce51a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=429, earliestPutTs=1680062065722 2023-03-29 03:54:26,072 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:54:26,072 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:54:26,072 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:54:26,072 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:26,072 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:54:26,080 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:54:26,080 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:54:26,080 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. 2023-03-29 03:54:26,080 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. started... 2023-03-29 03:54:26,080 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ee0c8ff38ef29b37b45871dfc89897f0 3/3 column families, dataSize=134.18 KB heapSize=366.23 KB 2023-03-29 03:54:26,081 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee0c8ff38ef29b37b45871dfc89897f0, store=A 2023-03-29 03:54:26,081 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-29 03:54:26,081 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee0c8ff38ef29b37b45871dfc89897f0, store=B 2023-03-29 03:54:26,081 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-29 03:54:26,081 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee0c8ff38ef29b37b45871dfc89897f0, store=C 2023-03-29 03:54:26,081 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-29 03:54:26,083 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] throttle.PressureAwareThroughputController(145): ee0c8ff38ef29b37b45871dfc89897f0#C#compaction#195 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:54:26,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on ee0c8ff38ef29b37b45871dfc89897f0 2023-03-29 03:54:26,119 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. as already flushing 2023-03-29 03:54:26,122 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303291c91e59956264bb196bffe9a54668f0f to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303291c91e59956264bb196bffe9a54668f0f 2023-03-29 03:54:26,132 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:26,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40148 deadline: 1680062126130, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:26,135 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:26,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40120 deadline: 1680062126132, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:26,138 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:26,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40142 deadline: 1680062126133, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:26,235 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:26,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40148 deadline: 1680062126233, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:26,240 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:26,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40120 deadline: 1680062126236, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:26,241 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:26,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40142 deadline: 1680062126239, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:26,439 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:26,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40148 deadline: 1680062126437, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:26,442 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:26,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40142 deadline: 1680062126442, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:26,444 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:26,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40120 deadline: 1680062126442, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:26,519 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/C/a3a629d6a981463c912ac67e9b5bad97 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/a3a629d6a981463c912ac67e9b5bad97 2023-03-29 03:54:26,524 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in ee0c8ff38ef29b37b45871dfc89897f0/C of ee0c8ff38ef29b37b45871dfc89897f0 into a3a629d6a981463c912ac67e9b5bad97(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:54:26,524 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ee0c8ff38ef29b37b45871dfc89897f0: 2023-03-29 03:54:26,524 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0., storeName=ee0c8ff38ef29b37b45871dfc89897f0/C, priority=10, startTime=1680062065958; duration=0sec 2023-03-29 03:54:26,524 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:54:26,533 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=463, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/A/03870896a22c467ca5b5c53f7b8dfca7 2023-03-29 03:54:26,552 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=463 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/B/edb0848871f547e795a3ac5c2940f746 2023-03-29 03:54:26,579 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=463 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/C/9da409063e004b1fa5f88ca35e8f6fcb 2023-03-29 03:54:26,583 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/A/03870896a22c467ca5b5c53f7b8dfca7 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/03870896a22c467ca5b5c53f7b8dfca7 2023-03-29 03:54:26,587 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/03870896a22c467ca5b5c53f7b8dfca7, entries=150, sequenceid=463, filesize=25.4 K 2023-03-29 03:54:26,588 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/B/edb0848871f547e795a3ac5c2940f746 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/edb0848871f547e795a3ac5c2940f746 2023-03-29 03:54:26,593 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/edb0848871f547e795a3ac5c2940f746, entries=150, sequenceid=463, filesize=11.9 K 2023-03-29 03:54:26,593 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/C/9da409063e004b1fa5f88ca35e8f6fcb as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/9da409063e004b1fa5f88ca35e8f6fcb 2023-03-29 03:54:26,597 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/9da409063e004b1fa5f88ca35e8f6fcb, entries=150, sequenceid=463, filesize=11.9 K 2023-03-29 03:54:26,598 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~366.19 KB/374976, currentSize=53.67 KB/54960 for ee0c8ff38ef29b37b45871dfc89897f0 in 517ms, sequenceid=463, compaction requested=false 2023-03-29 03:54:26,598 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ee0c8ff38ef29b37b45871dfc89897f0: 2023-03-29 03:54:26,598 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. 2023-03-29 03:54:26,598 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:54:26,598 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:54:26,598 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:26,598 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:54:26,598 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:54:26,599 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:26,599 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:26,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:26,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:26,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:26,600 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:26,600 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:54:26,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:26,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:26,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:26,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:26,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:26,601 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:54:26,601 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@188d97bd[Count = 0] remaining members to acquire global barrier 2023-03-29 03:54:26,601 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:54:26,601 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:26,602 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:26,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:26,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:26,603 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:54:26,603 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:54:26,603 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:54:26,603 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:26,603 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:54:26,605 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:26,605 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:54:26,605 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:26,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:26,605 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:54:26,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:26,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:26,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:26,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:26,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:26,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:26,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:26,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:26,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:26,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:54:26,608 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:54:26,608 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:54:26,608 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:54:26,608 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:54:26,608 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:54:26,608 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:54:26,610 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:26,610 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:26,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:26,610 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:26,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:26,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:26,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:26,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:26,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:26,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:26,610 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:26,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:26,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:26,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:26,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:26,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:26,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:26,612 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:26,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:26,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:26,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:26,616 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:26,616 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:26,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:26,616 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:26,616 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:26,616 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:26,616 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:26,616 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:26,616 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:54:26,616 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:54:26,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:26,616 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:26,617 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:26,617 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:26,618 DEBUG [Thread-985] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:54:26,618 DEBUG [Thread-985] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:54:26,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:26,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:26,718 DEBUG [Thread-985] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:54:26,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:54:26,722 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:54:26,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:54:26,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:54:26,723 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:54:26,723 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:26,723 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:54:26,724 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:54:26,724 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:26,724 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:26,731 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:26,731 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:26,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:26,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:26,732 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:26,732 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:54:26,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:26,732 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:26,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:54:26,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:26,733 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:54:26,733 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:54:26,733 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:54:26,733 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:54:26,733 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:26,734 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:54:26,734 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:54:26,734 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:54:26,734 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. 2023-03-29 03:54:26,734 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. started... 2023-03-29 03:54:26,734 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ee0c8ff38ef29b37b45871dfc89897f0 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-29 03:54:26,734 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee0c8ff38ef29b37b45871dfc89897f0, store=A 2023-03-29 03:54:26,734 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:54:26,734 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee0c8ff38ef29b37b45871dfc89897f0, store=B 2023-03-29 03:54:26,734 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:54:26,734 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee0c8ff38ef29b37b45871dfc89897f0, store=C 2023-03-29 03:54:26,734 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:54:26,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on ee0c8ff38ef29b37b45871dfc89897f0 2023-03-29 03:54:26,743 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. as already flushing 2023-03-29 03:54:26,759 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303292d4d17beae214d899c9c1e28710823d0 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303292d4d17beae214d899c9c1e28710823d0 2023-03-29 03:54:26,768 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680062066756, max timestamp=1680062066767Num uniques 859; 2023-03-29 03:54:26,768 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-29 03:54:26,768 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680062066756, max timestamp=1680062066767Num uniques 859; 2023-03-29 03:54:26,769 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-29 03:54:26,769 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680062066756, max timestamp=1680062066767Num uniques 859; 2023-03-29 03:54:26,769 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-29 03:54:26,786 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:26,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40142 deadline: 1680062126784, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:26,786 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:26,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 254 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40120 deadline: 1680062126785, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:26,789 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:26,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 254 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40148 deadline: 1680062126786, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:26,879 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/A/36d6fe4241854e6183136f1efaaf5468 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/36d6fe4241854e6183136f1efaaf5468 2023-03-29 03:54:26,884 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in ee0c8ff38ef29b37b45871dfc89897f0/A of ee0c8ff38ef29b37b45871dfc89897f0 into 36d6fe4241854e6183136f1efaaf5468(size=26.2 K), total size for store is 51.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:54:26,884 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ee0c8ff38ef29b37b45871dfc89897f0: 2023-03-29 03:54:26,884 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0., storeName=ee0c8ff38ef29b37b45871dfc89897f0/A, priority=10, startTime=1680062065958; duration=0sec 2023-03-29 03:54:26,884 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:54:26,888 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:26,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40142 deadline: 1680062126887, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:26,888 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:26,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 256 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40120 deadline: 1680062126887, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:26,890 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:26,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 256 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40148 deadline: 1680062126890, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:27,092 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:27,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 258 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40120 deadline: 1680062127089, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:27,093 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:27,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40142 deadline: 1680062127090, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:27,093 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:27,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 258 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40148 deadline: 1680062127092, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:27,166 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=475, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/A/7889044ff9464ebcbfecf8c5e2d96c9d 2023-03-29 03:54:27,178 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=475 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/B/482e057972164ba088f55f944e170dac 2023-03-29 03:54:27,194 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=475 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/C/9911eba33ab64624aae71a78afee6e15 2023-03-29 03:54:27,198 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/A/7889044ff9464ebcbfecf8c5e2d96c9d as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/7889044ff9464ebcbfecf8c5e2d96c9d 2023-03-29 03:54:27,202 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/7889044ff9464ebcbfecf8c5e2d96c9d, entries=150, sequenceid=475, filesize=25.4 K 2023-03-29 03:54:27,203 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/B/482e057972164ba088f55f944e170dac as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/482e057972164ba088f55f944e170dac 2023-03-29 03:54:27,208 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/482e057972164ba088f55f944e170dac, entries=150, sequenceid=475, filesize=11.9 K 2023-03-29 03:54:27,209 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/C/9911eba33ab64624aae71a78afee6e15 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/9911eba33ab64624aae71a78afee6e15 2023-03-29 03:54:27,213 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/9911eba33ab64624aae71a78afee6e15, entries=150, sequenceid=475, filesize=11.9 K 2023-03-29 03:54:27,214 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=134.18 KB/137400 for ee0c8ff38ef29b37b45871dfc89897f0 in 479ms, sequenceid=475, compaction requested=true 2023-03-29 03:54:27,214 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ee0c8ff38ef29b37b45871dfc89897f0: 2023-03-29 03:54:27,214 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. 2023-03-29 03:54:27,214 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:54:27,214 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:54:27,214 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:27,214 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:54:27,214 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:54:27,215 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:27,215 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:27,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:27,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:27,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:27,216 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:27,216 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:54:27,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:27,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:27,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:27,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:27,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:27,217 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:54:27,218 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@578cbe4e[Count = 0] remaining members to acquire global barrier 2023-03-29 03:54:27,218 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:54:27,218 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:27,219 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:27,219 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:27,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:27,219 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:54:27,219 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:54:27,219 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:54:27,220 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:27,220 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:54:27,222 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:27,222 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:54:27,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:27,222 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:27,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:27,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:27,222 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:54:27,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:27,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:27,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:27,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:27,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:27,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:27,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:27,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:54:27,225 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:54:27,225 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:54:27,225 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:54:27,225 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:54:27,225 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:54:27,225 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:54:27,227 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:27,227 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:27,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:27,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:27,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:27,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:27,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:27,228 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:27,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:27,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:27,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:27,228 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:27,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:27,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:27,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:27,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:27,230 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:27,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:27,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:27,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:27,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:27,233 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:27,233 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:27,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:27,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:27,233 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:54:27,233 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:54:27,233 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:54:27,233 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:27,234 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:27,234 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:27,234 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:27,234 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:27,234 DEBUG [Thread-985] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:54:27,233 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:27,234 DEBUG [Thread-985] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:54:27,234 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:27,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:27,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:27,234 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:27,334 DEBUG [Thread-985] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:54:27,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:54:27,336 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:54:27,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:54:27,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:54:27,338 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:54:27,338 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:27,338 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:54:27,338 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:54:27,338 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:27,338 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:27,339 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:27,339 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:27,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:27,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:27,340 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:27,340 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:54:27,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:27,340 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:27,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:54:27,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:27,341 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:54:27,341 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:54:27,341 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:54:27,341 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:54:27,341 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:27,341 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:54:27,342 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:54:27,342 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:54:27,342 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. 2023-03-29 03:54:27,342 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. started... 2023-03-29 03:54:27,342 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ee0c8ff38ef29b37b45871dfc89897f0 3/3 column families, dataSize=134.18 KB heapSize=370.92 KB 2023-03-29 03:54:27,342 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee0c8ff38ef29b37b45871dfc89897f0, store=A 2023-03-29 03:54:27,342 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-29 03:54:27,342 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee0c8ff38ef29b37b45871dfc89897f0, store=B 2023-03-29 03:54:27,343 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-29 03:54:27,343 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee0c8ff38ef29b37b45871dfc89897f0, store=C 2023-03-29 03:54:27,343 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-29 03:54:27,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on ee0c8ff38ef29b37b45871dfc89897f0 2023-03-29 03:54:27,396 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. as already flushing 2023-03-29 03:54:27,410 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:27,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 262 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40148 deadline: 1680062127406, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:27,410 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:27,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 263 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40120 deadline: 1680062127407, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:27,410 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:27,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 256 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40142 deadline: 1680062127407, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:27,516 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:27,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 264 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40148 deadline: 1680062127511, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:27,516 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:27,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 265 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40120 deadline: 1680062127511, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:27,517 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:27,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 258 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40142 deadline: 1680062127511, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:27,721 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:27,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 266 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40148 deadline: 1680062127717, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:27,721 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:27,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 260 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40142 deadline: 1680062127717, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:27,721 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:27,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 267 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40120 deadline: 1680062127718, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:27,756 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230329dee91d61916c44d9ad0d27e0cf1e8965 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230329dee91d61916c44d9ad0d27e0cf1e8965 2023-03-29 03:54:28,024 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:28,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 269 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40120 deadline: 1680062128022, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:28,024 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:28,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 268 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40148 deadline: 1680062128023, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:28,024 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:28,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 262 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40142 deadline: 1680062128023, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:28,162 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=510, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/A/ee989dac96ba435993730a3b8609b30e 2023-03-29 03:54:28,529 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:28,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 270 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40148 deadline: 1680062128528, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:28,529 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:28,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 264 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40142 deadline: 1680062128528, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:28,530 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:28,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 271 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40120 deadline: 1680062128529, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:28,584 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=510 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/B/b8f06cbe19994d249946f31238e303a6 2023-03-29 03:54:28,728 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:28,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40124 deadline: 1680062128724, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:28,730 DEBUG [Thread-983] client.RpcRetryingCallerImpl(129): Call exception, tries=8, retries=16, started=18238 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0., hostname=jenkins-hbase4.apache.org,35599,1680062018534, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-29 03:54:28,756 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:28,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40112 deadline: 1680062128756, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:28,758 DEBUG [Thread-979] client.RpcRetryingCallerImpl(129): Call exception, tries=8, retries=16, started=18267 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0., hostname=jenkins-hbase4.apache.org,35599,1680062018534, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-29 03:54:28,997 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=510 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/C/418786d53267494da51baa6c756647dd 2023-03-29 03:54:29,002 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/A/ee989dac96ba435993730a3b8609b30e as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/ee989dac96ba435993730a3b8609b30e 2023-03-29 03:54:29,007 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/ee989dac96ba435993730a3b8609b30e, entries=150, sequenceid=510, filesize=25.4 K 2023-03-29 03:54:29,008 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/B/b8f06cbe19994d249946f31238e303a6 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/b8f06cbe19994d249946f31238e303a6 2023-03-29 03:54:29,012 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/b8f06cbe19994d249946f31238e303a6, entries=150, sequenceid=510, filesize=11.9 K 2023-03-29 03:54:29,013 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/C/418786d53267494da51baa6c756647dd as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/418786d53267494da51baa6c756647dd 2023-03-29 03:54:29,018 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/418786d53267494da51baa6c756647dd, entries=150, sequenceid=510, filesize=11.9 K 2023-03-29 03:54:29,019 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~370.88 KB/379776, currentSize=53.67 KB/54960 for ee0c8ff38ef29b37b45871dfc89897f0 in 1677ms, sequenceid=510, compaction requested=true 2023-03-29 03:54:29,019 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ee0c8ff38ef29b37b45871dfc89897f0: 2023-03-29 03:54:29,019 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. 2023-03-29 03:54:29,019 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:54:29,019 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:54:29,019 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:29,019 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:54:29,019 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:54:29,021 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:29,021 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:29,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:29,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:29,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:29,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:29,022 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:29,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:29,022 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:54:29,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:29,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:29,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:29,024 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:54:29,024 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@172a8cf4[Count = 0] remaining members to acquire global barrier 2023-03-29 03:54:29,024 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:54:29,024 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:29,026 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:29,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:29,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:29,026 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:54:29,026 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:54:29,026 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:54:29,026 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:29,027 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:54:29,027 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:54:29,028 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:29,027 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:29,028 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:54:29,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:29,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:29,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:29,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:29,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:29,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:29,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:29,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:29,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:29,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:29,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:54:29,032 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:54:29,032 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:54:29,032 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:54:29,032 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:54:29,032 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:54:29,032 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:54:29,034 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:29,034 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:29,035 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:29,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:29,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:29,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:29,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:29,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:29,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:29,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:29,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:29,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:29,036 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:29,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:29,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:29,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:29,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:29,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:29,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:29,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:29,038 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:29,040 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:29,040 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:29,040 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:29,040 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:29,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:29,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:29,040 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:29,040 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:29,040 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:29,040 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:54:29,041 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:54:29,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:29,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:29,041 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:29,041 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:29,041 DEBUG [Thread-985] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:54:29,041 DEBUG [Thread-985] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:54:29,041 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:29,141 DEBUG [Thread-985] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:54:29,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:54:29,143 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:54:29,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:54:29,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:54:29,145 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:54:29,145 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:29,145 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:54:29,145 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:54:29,145 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:29,145 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:29,146 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:29,146 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:29,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:29,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:29,147 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:29,147 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:54:29,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:29,147 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:29,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:54:29,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:29,148 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:54:29,148 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:54:29,148 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:54:29,148 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:54:29,148 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:29,148 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:54:29,149 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:54:29,149 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:54:29,149 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. 2023-03-29 03:54:29,149 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. started... 2023-03-29 03:54:29,149 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ee0c8ff38ef29b37b45871dfc89897f0 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-29 03:54:29,149 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee0c8ff38ef29b37b45871dfc89897f0, store=A 2023-03-29 03:54:29,149 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:54:29,149 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee0c8ff38ef29b37b45871dfc89897f0, store=B 2023-03-29 03:54:29,149 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:54:29,149 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee0c8ff38ef29b37b45871dfc89897f0, store=C 2023-03-29 03:54:29,150 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:54:29,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on ee0c8ff38ef29b37b45871dfc89897f0 2023-03-29 03:54:29,539 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. as already flushing 2023-03-29 03:54:29,566 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230329b1264c03ece04817bee46fb092e80be3 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230329b1264c03ece04817bee46fb092e80be3 2023-03-29 03:54:29,567 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1680062069553, max timestamp=1680062069565Num uniques 859; 2023-03-29 03:54:29,567 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-03-29 03:54:29,567 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1680062069553, max timestamp=1680062069565Num uniques 859; 2023-03-29 03:54:29,567 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1680062069553, max timestamp=1680062069565Num uniques 859; 2023-03-29 03:54:29,567 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-03-29 03:54:29,567 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-03-29 03:54:29,585 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:29,585 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:29,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 276 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40142 deadline: 1680062129582, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:29,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 283 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40120 deadline: 1680062129583, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:29,585 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:29,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 282 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40148 deadline: 1680062129584, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:29,687 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:29,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 278 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40142 deadline: 1680062129686, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:29,688 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:29,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 284 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40148 deadline: 1680062129687, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:29,688 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:29,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 285 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40120 deadline: 1680062129687, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:29,890 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:29,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 280 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40142 deadline: 1680062129890, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:29,890 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:29,891 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:29,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 286 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40148 deadline: 1680062129890, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:29,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 287 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40120 deadline: 1680062129890, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:29,973 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=521, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/A/840458d7ead84332b813b26368cf6098 2023-03-29 03:54:29,986 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=521 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/B/1b7af5cd63364cabb3e0978af7b359c4 2023-03-29 03:54:29,999 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=521 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/C/b093b0efc8f545c6bf91b8cd62d78d2a 2023-03-29 03:54:30,003 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/A/840458d7ead84332b813b26368cf6098 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/840458d7ead84332b813b26368cf6098 2023-03-29 03:54:30,007 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/840458d7ead84332b813b26368cf6098, entries=150, sequenceid=521, filesize=25.4 K 2023-03-29 03:54:30,008 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/B/1b7af5cd63364cabb3e0978af7b359c4 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/1b7af5cd63364cabb3e0978af7b359c4 2023-03-29 03:54:30,012 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/1b7af5cd63364cabb3e0978af7b359c4, entries=150, sequenceid=521, filesize=11.9 K 2023-03-29 03:54:30,013 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/C/b093b0efc8f545c6bf91b8cd62d78d2a as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/b093b0efc8f545c6bf91b8cd62d78d2a 2023-03-29 03:54:30,016 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/b093b0efc8f545c6bf91b8cd62d78d2a, entries=150, sequenceid=521, filesize=11.9 K 2023-03-29 03:54:30,017 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=134.18 KB/137400 for ee0c8ff38ef29b37b45871dfc89897f0 in 868ms, sequenceid=521, compaction requested=true 2023-03-29 03:54:30,018 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ee0c8ff38ef29b37b45871dfc89897f0: 2023-03-29 03:54:30,018 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. 2023-03-29 03:54:30,018 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:54:30,018 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:54:30,018 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:30,018 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:54:30,018 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:54:30,020 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:30,020 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:30,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:30,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:30,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:30,020 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:30,020 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:54:30,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:30,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:30,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:30,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:30,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:30,022 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:54:30,022 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@13c13067[Count = 0] remaining members to acquire global barrier 2023-03-29 03:54:30,022 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:54:30,022 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:30,024 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:30,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:30,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:30,024 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:54:30,024 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:54:30,024 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:54:30,025 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:30,025 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:54:30,026 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:30,026 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:54:30,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:30,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:30,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:30,026 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:30,026 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:54:30,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:30,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:30,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:30,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:30,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:30,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:30,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:30,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:54:30,028 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:54:30,028 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:54:30,028 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:54:30,028 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:54:30,028 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:54:30,028 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:54:30,030 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:30,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:30,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:30,030 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:30,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:30,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:30,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:30,030 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:30,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:30,031 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:30,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:30,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:30,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:30,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:30,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:30,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:30,032 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:30,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:30,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:30,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:30,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:30,034 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:30,034 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:30,034 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:30,035 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:30,034 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:30,035 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:54:30,035 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:54:30,035 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:30,035 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:54:30,035 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:30,035 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:30,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:30,035 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:30,035 DEBUG [Thread-985] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:54:30,035 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:30,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:30,035 DEBUG [Thread-985] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:54:30,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:30,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:30,135 DEBUG [Thread-985] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:54:30,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:54:30,137 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:54:30,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:54:30,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:54:30,139 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:54:30,139 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:30,139 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:54:30,139 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:54:30,139 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:30,139 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:30,142 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:30,142 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:30,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:30,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:30,142 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:30,142 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:54:30,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:30,142 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:30,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:54:30,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:30,143 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:54:30,143 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:54:30,143 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:54:30,143 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:54:30,143 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:30,143 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:54:30,143 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:54:30,143 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:54:30,143 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. 2023-03-29 03:54:30,143 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. started... 2023-03-29 03:54:30,144 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing ee0c8ff38ef29b37b45871dfc89897f0 3/3 column families, dataSize=134.18 KB heapSize=366.23 KB 2023-03-29 03:54:30,144 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee0c8ff38ef29b37b45871dfc89897f0, store=A 2023-03-29 03:54:30,144 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-29 03:54:30,144 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee0c8ff38ef29b37b45871dfc89897f0, store=B 2023-03-29 03:54:30,144 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-29 03:54:30,144 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee0c8ff38ef29b37b45871dfc89897f0, store=C 2023-03-29 03:54:30,144 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-29 03:54:30,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on ee0c8ff38ef29b37b45871dfc89897f0 2023-03-29 03:54:30,195 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. as already flushing 2023-03-29 03:54:30,221 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:30,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 292 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40120 deadline: 1680062130216, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:30,221 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:30,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 284 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40142 deadline: 1680062130217, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:30,224 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:30,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 291 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40148 deadline: 1680062130221, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:30,322 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:30,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 294 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40120 deadline: 1680062130322, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:30,325 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:30,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 286 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40142 deadline: 1680062130323, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:30,327 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:30,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 293 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40148 deadline: 1680062130325, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:30,347 DEBUG [Thread-992] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x42d325f4 to 127.0.0.1:52103 2023-03-29 03:54:30,347 DEBUG [Thread-992] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-29 03:54:30,348 DEBUG [Thread-988] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4a0fd7b4 to 127.0.0.1:52103 2023-03-29 03:54:30,348 DEBUG [Thread-988] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-29 03:54:30,350 DEBUG [Thread-986] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x421e5356 to 127.0.0.1:52103 2023-03-29 03:54:30,350 DEBUG [Thread-986] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-29 03:54:30,353 DEBUG [Thread-990] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1a7b66a3 to 127.0.0.1:52103 2023-03-29 03:54:30,353 DEBUG [Thread-990] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-29 03:54:30,525 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:30,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 296 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40120 deadline: 1680062130525, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:30,527 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:30,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 288 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40142 deadline: 1680062130527, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:30,529 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:30,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 295 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40148 deadline: 1680062130529, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:30,559 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230329cd7affa96cb3456b9fea1ebe36999263 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230329cd7affa96cb3456b9fea1ebe36999263 2023-03-29 03:54:30,563 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=554, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/A/3c65c3c9e46b462c9a4702df8ae462e2 2023-03-29 03:54:30,826 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:30,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 298 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40120 deadline: 1680062130826, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:30,829 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:30,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 290 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40142 deadline: 1680062130829, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:30,832 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:30,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 297 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40148 deadline: 1680062130832, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:30,972 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=554 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/B/027c411e7f2a4d1fa6901eb675441ca1 2023-03-29 03:54:31,330 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:31,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 300 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40120 deadline: 1680062131329, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:31,331 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:31,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 292 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40142 deadline: 1680062131331, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:31,336 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:31,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 299 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40148 deadline: 1680062131336, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:31,382 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=554 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/C/3599483f9cad42e4b5c15bcc3f90b48a 2023-03-29 03:54:31,386 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/A/3c65c3c9e46b462c9a4702df8ae462e2 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/3c65c3c9e46b462c9a4702df8ae462e2 2023-03-29 03:54:31,389 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/3c65c3c9e46b462c9a4702df8ae462e2, entries=150, sequenceid=554, filesize=25.4 K 2023-03-29 03:54:31,390 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/B/027c411e7f2a4d1fa6901eb675441ca1 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/027c411e7f2a4d1fa6901eb675441ca1 2023-03-29 03:54:31,394 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/027c411e7f2a4d1fa6901eb675441ca1, entries=150, sequenceid=554, filesize=11.9 K 2023-03-29 03:54:31,395 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/C/3599483f9cad42e4b5c15bcc3f90b48a as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/3599483f9cad42e4b5c15bcc3f90b48a 2023-03-29 03:54:31,399 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/3599483f9cad42e4b5c15bcc3f90b48a, entries=150, sequenceid=554, filesize=11.9 K 2023-03-29 03:54:31,400 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~366.19 KB/374976, currentSize=53.67 KB/54960 for ee0c8ff38ef29b37b45871dfc89897f0 in 1257ms, sequenceid=554, compaction requested=true 2023-03-29 03:54:31,400 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for ee0c8ff38ef29b37b45871dfc89897f0: 2023-03-29 03:54:31,400 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. 2023-03-29 03:54:31,400 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:54:31,400 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:54:31,400 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:31,400 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:54:31,400 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:54:31,404 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:31,404 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:31,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:31,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:31,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:31,405 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:31,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:31,405 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:54:31,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:31,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:31,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:31,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:31,406 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:54:31,406 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@67856ddd[Count = 0] remaining members to acquire global barrier 2023-03-29 03:54:31,406 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:54:31,406 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:31,407 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:31,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:31,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:31,407 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:54:31,407 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:54:31,407 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:54:31,407 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:31,407 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:54:31,409 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:54:31,409 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:31,409 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:31,409 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:54:31,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:31,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:31,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:31,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:31,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:31,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:31,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:31,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:31,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:31,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:31,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:54:31,412 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:54:31,412 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:54:31,412 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:54:31,412 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:54:31,412 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:54:31,412 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:54:31,414 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:31,414 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:31,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:31,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:31,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:31,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:31,414 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:31,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:31,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:31,414 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:31,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:31,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:31,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:31,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:31,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:31,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:31,416 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:31,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:31,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:31,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:31,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:31,418 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:31,418 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:31,418 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:31,418 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:31,418 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:54:31,418 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:54:31,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:31,418 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:31,418 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:31,418 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:31,418 DEBUG [Thread-985] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:54:31,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:31,418 DEBUG [Thread-985] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:54:31,418 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:31,418 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:31,418 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:31,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:31,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:31,518 DEBUG [Thread-985] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:54:31,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:54:32,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on ee0c8ff38ef29b37b45871dfc89897f0 2023-03-29 03:54:32,340 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing ee0c8ff38ef29b37b45871dfc89897f0 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-29 03:54:32,340 DEBUG [Thread-981] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x210d1008 to 127.0.0.1:52103 2023-03-29 03:54:32,340 DEBUG [Thread-981] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-29 03:54:32,340 DEBUG [Thread-977] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x24bd3872 to 127.0.0.1:52103 2023-03-29 03:54:32,340 DEBUG [Thread-977] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-29 03:54:32,341 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee0c8ff38ef29b37b45871dfc89897f0, store=A 2023-03-29 03:54:32,341 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:54:32,341 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee0c8ff38ef29b37b45871dfc89897f0, store=B 2023-03-29 03:54:32,341 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:54:32,341 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee0c8ff38ef29b37b45871dfc89897f0, store=C 2023-03-29 03:54:32,341 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:54:32,341 DEBUG [Thread-975] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x55cbb115 to 127.0.0.1:52103 2023-03-29 03:54:32,341 DEBUG [Thread-975] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-29 03:54:32,757 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032989e1b96c544e41ddaf1774bdc941750f to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032989e1b96c544e41ddaf1774bdc941750f 2023-03-29 03:54:33,162 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=568, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/A/4c7ab22bf037435cac07e549ca98ed26 2023-03-29 03:54:33,573 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=568 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/B/4e0cff8b04d544758103e72d560e3630 2023-03-29 03:54:33,982 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=568 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/C/3a48a2d6b83f4b79b4e185d95cc816f4 2023-03-29 03:54:33,986 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/A/4c7ab22bf037435cac07e549ca98ed26 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/4c7ab22bf037435cac07e549ca98ed26 2023-03-29 03:54:33,990 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/4c7ab22bf037435cac07e549ca98ed26, entries=150, sequenceid=568, filesize=25.4 K 2023-03-29 03:54:33,991 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/B/4e0cff8b04d544758103e72d560e3630 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/4e0cff8b04d544758103e72d560e3630 2023-03-29 03:54:33,994 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/4e0cff8b04d544758103e72d560e3630, entries=150, sequenceid=568, filesize=11.9 K 2023-03-29 03:54:33,995 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/C/3a48a2d6b83f4b79b4e185d95cc816f4 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/3a48a2d6b83f4b79b4e185d95cc816f4 2023-03-29 03:54:33,998 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/3a48a2d6b83f4b79b4e185d95cc816f4, entries=150, sequenceid=568, filesize=11.9 K 2023-03-29 03:54:33,999 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=0 B/0 for ee0c8ff38ef29b37b45871dfc89897f0 in 1659ms, sequenceid=568, compaction requested=true 2023-03-29 03:54:33,999 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for ee0c8ff38ef29b37b45871dfc89897f0: 2023-03-29 03:54:33,999 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:54:33,999 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:54:33,999 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-03-29 03:54:33,999 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-03-29 03:54:33,999 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:54:34,001 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 183106 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-03-29 03:54:34,001 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 86086 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-03-29 03:54:34,001 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1912): ee0c8ff38ef29b37b45871dfc89897f0/A is initiating minor compaction (all files) 2023-03-29 03:54:34,001 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1912): ee0c8ff38ef29b37b45871dfc89897f0/B is initiating minor compaction (all files) 2023-03-29 03:54:34,001 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of ee0c8ff38ef29b37b45871dfc89897f0/A in TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. 2023-03-29 03:54:34,001 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ee0c8ff38ef29b37b45871dfc89897f0/B in TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. 2023-03-29 03:54:34,001 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/36d6fe4241854e6183136f1efaaf5468, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/03870896a22c467ca5b5c53f7b8dfca7, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/7889044ff9464ebcbfecf8c5e2d96c9d, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/ee989dac96ba435993730a3b8609b30e, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/840458d7ead84332b813b26368cf6098, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/3c65c3c9e46b462c9a4702df8ae462e2, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/4c7ab22bf037435cac07e549ca98ed26] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp, totalSize=178.8 K 2023-03-29 03:54:34,002 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/d85df311c0f349299d2007f6b7007c13, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/edb0848871f547e795a3ac5c2940f746, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/482e057972164ba088f55f944e170dac, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/b8f06cbe19994d249946f31238e303a6, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/1b7af5cd63364cabb3e0978af7b359c4, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/027c411e7f2a4d1fa6901eb675441ca1, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/4e0cff8b04d544758103e72d560e3630] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp, totalSize=84.1 K 2023-03-29 03:54:34,002 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 36d6fe4241854e6183136f1efaaf5468, keycount=150, bloomtype=ROW, size=26.2 K, encoding=NONE, compression=NONE, seqNum=429, earliestPutTs=1680062065722 2023-03-29 03:54:34,002 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting d85df311c0f349299d2007f6b7007c13, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=429, earliestPutTs=1680062065722 2023-03-29 03:54:34,002 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 03870896a22c467ca5b5c53f7b8dfca7, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=463, earliestPutTs=1680062065803 2023-03-29 03:54:34,002 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting edb0848871f547e795a3ac5c2940f746, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=463, earliestPutTs=1680062065803 2023-03-29 03:54:34,002 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 7889044ff9464ebcbfecf8c5e2d96c9d, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=475, earliestPutTs=1680062066126 2023-03-29 03:54:34,003 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 482e057972164ba088f55f944e170dac, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=475, earliestPutTs=1680062066126 2023-03-29 03:54:34,003 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting ee989dac96ba435993730a3b8609b30e, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=510, earliestPutTs=1680062066783 2023-03-29 03:54:34,003 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting b8f06cbe19994d249946f31238e303a6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=510, earliestPutTs=1680062066783 2023-03-29 03:54:34,003 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 840458d7ead84332b813b26368cf6098, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=521, earliestPutTs=1680062067405 2023-03-29 03:54:34,003 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 1b7af5cd63364cabb3e0978af7b359c4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=521, earliestPutTs=1680062067405 2023-03-29 03:54:34,003 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 3c65c3c9e46b462c9a4702df8ae462e2, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=554, earliestPutTs=1680062069578 2023-03-29 03:54:34,003 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 027c411e7f2a4d1fa6901eb675441ca1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=554, earliestPutTs=1680062069578 2023-03-29 03:54:34,004 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 4c7ab22bf037435cac07e549ca98ed26, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=568, earliestPutTs=1680062070214 2023-03-29 03:54:34,004 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 4e0cff8b04d544758103e72d560e3630, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=568, earliestPutTs=1680062070214 2023-03-29 03:54:34,016 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] throttle.PressureAwareThroughputController(145): ee0c8ff38ef29b37b45871dfc89897f0#B#compaction#215 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:54:34,018 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] throttle.PressureAwareThroughputController(145): ee0c8ff38ef29b37b45871dfc89897f0#A#compaction#214 average throughput is 4.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:54:34,425 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/B/8e005e41e7574d39a03fb3c055e72646 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/8e005e41e7574d39a03fb3c055e72646 2023-03-29 03:54:34,430 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in ee0c8ff38ef29b37b45871dfc89897f0/B of ee0c8ff38ef29b37b45871dfc89897f0 into 8e005e41e7574d39a03fb3c055e72646(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:54:34,430 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ee0c8ff38ef29b37b45871dfc89897f0: 2023-03-29 03:54:34,430 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0., storeName=ee0c8ff38ef29b37b45871dfc89897f0/B, priority=9, startTime=1680062073999; duration=0sec 2023-03-29 03:54:34,430 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:54:34,430 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-03-29 03:54:34,432 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 86052 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-03-29 03:54:34,432 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1912): ee0c8ff38ef29b37b45871dfc89897f0/C is initiating minor compaction (all files) 2023-03-29 03:54:34,432 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2259): Starting compaction of ee0c8ff38ef29b37b45871dfc89897f0/C in TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. 2023-03-29 03:54:34,432 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/a3a629d6a981463c912ac67e9b5bad97, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/9da409063e004b1fa5f88ca35e8f6fcb, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/9911eba33ab64624aae71a78afee6e15, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/418786d53267494da51baa6c756647dd, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/b093b0efc8f545c6bf91b8cd62d78d2a, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/3599483f9cad42e4b5c15bcc3f90b48a, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/3a48a2d6b83f4b79b4e185d95cc816f4] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp, totalSize=84.0 K 2023-03-29 03:54:34,432 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting a3a629d6a981463c912ac67e9b5bad97, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=429, earliestPutTs=1680062065722 2023-03-29 03:54:34,433 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 9da409063e004b1fa5f88ca35e8f6fcb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=463, earliestPutTs=1680062065803 2023-03-29 03:54:34,433 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 9911eba33ab64624aae71a78afee6e15, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=475, earliestPutTs=1680062066126 2023-03-29 03:54:34,433 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 418786d53267494da51baa6c756647dd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=510, earliestPutTs=1680062066783 2023-03-29 03:54:34,433 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting b093b0efc8f545c6bf91b8cd62d78d2a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=521, earliestPutTs=1680062067405 2023-03-29 03:54:34,434 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 3599483f9cad42e4b5c15bcc3f90b48a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=554, earliestPutTs=1680062069578 2023-03-29 03:54:34,434 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 3a48a2d6b83f4b79b4e185d95cc816f4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=568, earliestPutTs=1680062070214 2023-03-29 03:54:34,458 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] throttle.PressureAwareThroughputController(145): ee0c8ff38ef29b37b45871dfc89897f0#C#compaction#216 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:54:34,867 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/C/5fe2a8cf6e754b26a81a0e54d35e9262 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/5fe2a8cf6e754b26a81a0e54d35e9262 2023-03-29 03:54:34,871 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in ee0c8ff38ef29b37b45871dfc89897f0/C of ee0c8ff38ef29b37b45871dfc89897f0 into 5fe2a8cf6e754b26a81a0e54d35e9262(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:54:34,871 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for ee0c8ff38ef29b37b45871dfc89897f0: 2023-03-29 03:54:34,871 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0., storeName=ee0c8ff38ef29b37b45871dfc89897f0/C, priority=9, startTime=1680062073999; duration=0sec 2023-03-29 03:54:34,871 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:54:35,235 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/A/566d7c29b94d4ebb83418981213e736b as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/566d7c29b94d4ebb83418981213e736b 2023-03-29 03:54:35,239 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in ee0c8ff38ef29b37b45871dfc89897f0/A of ee0c8ff38ef29b37b45871dfc89897f0 into 566d7c29b94d4ebb83418981213e736b(size=26.4 K), total size for store is 26.4 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-03-29 03:54:35,239 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for ee0c8ff38ef29b37b45871dfc89897f0: 2023-03-29 03:54:35,239 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0., storeName=ee0c8ff38ef29b37b45871dfc89897f0/A, priority=9, startTime=1680062073999; duration=1sec 2023-03-29 03:54:35,239 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:54:38,793 DEBUG [Thread-983] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x00c2598b to 127.0.0.1:52103 2023-03-29 03:54:38,793 DEBUG [Thread-983] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-29 03:54:38,858 DEBUG [Thread-979] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7aba8e46 to 127.0.0.1:52103 2023-03-29 03:54:38,858 DEBUG [Thread-979] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-29 03:54:38,858 INFO [Listener at localhost/35027] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-03-29 03:54:38,858 INFO [Listener at localhost/35027] hbase.AcidGuaranteesTestTool(394): wrote 155 2023-03-29 03:54:38,858 INFO [Listener at localhost/35027] hbase.AcidGuaranteesTestTool(394): wrote 154 2023-03-29 03:54:38,859 INFO [Listener at localhost/35027] hbase.AcidGuaranteesTestTool(394): wrote 11 2023-03-29 03:54:38,859 INFO [Listener at localhost/35027] hbase.AcidGuaranteesTestTool(394): wrote 147 2023-03-29 03:54:38,859 INFO [Listener at localhost/35027] hbase.AcidGuaranteesTestTool(394): wrote 10 2023-03-29 03:54:38,859 INFO [Listener at localhost/35027] hbase.AcidGuaranteesTestTool(396): Readers: 2023-03-29 03:54:38,859 INFO [Listener at localhost/35027] hbase.AcidGuaranteesTestTool(398): read 4495 2023-03-29 03:54:38,859 INFO [Listener at localhost/35027] hbase.AcidGuaranteesTestTool(398): read 4465 2023-03-29 03:54:38,859 INFO [Listener at localhost/35027] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-03-29 03:54:38,859 INFO [Listener at localhost/35027] hbase.AcidGuaranteesTestTool(402): scanned 1919 2023-03-29 03:54:38,859 INFO [Listener at localhost/35027] hbase.AcidGuaranteesTestTool(403): verified 5757 rows 2023-03-29 03:54:38,859 INFO [Listener at localhost/35027] hbase.AcidGuaranteesTestTool(402): scanned 1919 2023-03-29 03:54:38,859 INFO [Listener at localhost/35027] hbase.AcidGuaranteesTestTool(403): verified 5757 rows 2023-03-29 03:54:38,859 INFO [Listener at localhost/35027] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-03-29 03:54:38,859 DEBUG [Listener at localhost/35027] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x324dc814 to 127.0.0.1:52103 2023-03-29 03:54:38,859 DEBUG [Listener at localhost/35027] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-29 03:54:38,860 INFO [Listener at localhost/35027] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-03-29 03:54:38,861 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-03-29 03:54:38,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure2.ProcedureExecutor(1029): Stored pid=24, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-03-29 03:54:38,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-03-29 03:54:38,865 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680062078865"}]},"ts":"1680062078865"} 2023-03-29 03:54:38,866 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-03-29 03:54:38,962 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-03-29 03:54:38,963 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=25, ppid=24, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=ee0c8ff38ef29b37b45871dfc89897f0, UNASSIGN}] 2023-03-29 03:54:38,965 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=25, ppid=24, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=ee0c8ff38ef29b37b45871dfc89897f0, UNASSIGN 2023-03-29 03:54:38,965 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=ee0c8ff38ef29b37b45871dfc89897f0, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:38,966 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680062078965"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680062078965"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680062078965"}]},"ts":"1680062078965"} 2023-03-29 03:54:38,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-03-29 03:54:38,967 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=26, ppid=25, state=RUNNABLE; CloseRegionProcedure ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534}] 2023-03-29 03:54:39,117 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close ee0c8ff38ef29b37b45871dfc89897f0 2023-03-29 03:54:39,117 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing ee0c8ff38ef29b37b45871dfc89897f0, disabling compactions & flushes 2023-03-29 03:54:39,117 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. 2023-03-29 03:54:39,117 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. 2023-03-29 03:54:39,117 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. after waiting 0 ms 2023-03-29 03:54:39,117 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. 2023-03-29 03:54:39,118 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing ee0c8ff38ef29b37b45871dfc89897f0 3/3 column families, dataSize=13.42 KB heapSize=40.59 KB 2023-03-29 03:54:39,118 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee0c8ff38ef29b37b45871dfc89897f0, store=A 2023-03-29 03:54:39,118 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:54:39,118 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee0c8ff38ef29b37b45871dfc89897f0, store=B 2023-03-29 03:54:39,118 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:54:39,118 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK ee0c8ff38ef29b37b45871dfc89897f0, store=C 2023-03-29 03:54:39,118 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:54:39,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-03-29 03:54:39,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-03-29 03:54:39,529 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303294b2d07f1d98844828806fee2b40fce57 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303294b2d07f1d98844828806fee2b40fce57 2023-03-29 03:54:39,880 DEBUG [regionserver/jenkins-hbase4:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.0, tune throughput to 50.00 MB/second 2023-03-29 03:54:39,935 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=576, memsize=4.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/A/9ca46251e46a469a9ce35c295ce6ff8c 2023-03-29 03:54:39,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-03-29 03:54:40,344 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=576 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/B/c7d173249d28446a8a0c44ef3b307438 2023-03-29 03:54:40,754 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=576 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/C/e0c5286f16de44849db233c5ba280e34 2023-03-29 03:54:40,758 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/A/9ca46251e46a469a9ce35c295ce6ff8c as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/9ca46251e46a469a9ce35c295ce6ff8c 2023-03-29 03:54:40,761 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/9ca46251e46a469a9ce35c295ce6ff8c, entries=50, sequenceid=576, filesize=11.7 K 2023-03-29 03:54:40,762 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/B/c7d173249d28446a8a0c44ef3b307438 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/c7d173249d28446a8a0c44ef3b307438 2023-03-29 03:54:40,765 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/c7d173249d28446a8a0c44ef3b307438, entries=50, sequenceid=576, filesize=7.1 K 2023-03-29 03:54:40,766 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/.tmp/C/e0c5286f16de44849db233c5ba280e34 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/e0c5286f16de44849db233c5ba280e34 2023-03-29 03:54:40,769 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/e0c5286f16de44849db233c5ba280e34, entries=50, sequenceid=576, filesize=7.1 K 2023-03-29 03:54:40,770 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~13.42 KB/13740, heapSize ~40.55 KB/41520, currentSize=0 B/0 for ee0c8ff38ef29b37b45871dfc89897f0 in 1653ms, sequenceid=576, compaction requested=false 2023-03-29 03:54:40,789 DEBUG [StoreCloser-TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/a8fc006fc607439e884193ed31339f0d, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/310be1c32da64f1295d33cad9bb2c787, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/2df21fa89e2847fdaa6e5c1978c4cf1e, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/9c01fbdb1bda447ab51e87c017b39dd9, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/9318058e9cb948d5b9aa3fd907a7ec82, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/da2a464998bf47e9a6082c65e09a7fa1, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/9beaaea60d37429d8490eeb284323256, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/e9f0af882d4846c99046a67a1818729b, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/ae356b7a53184a01ba044c066a5c5acb, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/2bd37e8188f44cc49ca55220c82a7025, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/fda0e2948cf147e9a1efaa7733436dad, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/3f0ed081b874469e8224777187f9d70a, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/0d5a970b176e497dbf41a2abbce4c4d4, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/7ecb6ce9ba7941ba80c3bed0abe1079d, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/3110413ad6634b3e8578f2e900123436, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/85008277797347258066bb667b5c6765, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/5d91b581dce24de0a3dda2eb48ebc725, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/f4c67b9d9ba04dc7bf95c7e502fbf7b8, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/84d423adfc6d4ffb98ddc2e54747c29d, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/48a0bdf086464a46b94c6707ecb26a38, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/7b25a644464b4ea8b5d49653d4f65ed8, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/d9cb6508c43d48328ac893ce7b94a47b, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/095f42e40c1b4d29bd6ad059f691b7ba, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/36d6fe4241854e6183136f1efaaf5468, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/03870896a22c467ca5b5c53f7b8dfca7, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/7889044ff9464ebcbfecf8c5e2d96c9d, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/ee989dac96ba435993730a3b8609b30e, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/840458d7ead84332b813b26368cf6098, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/3c65c3c9e46b462c9a4702df8ae462e2, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/4c7ab22bf037435cac07e549ca98ed26] to archive 2023-03-29 03:54:40,790 DEBUG [StoreCloser-TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-29 03:54:40,791 DEBUG [StoreCloser-TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/a8fc006fc607439e884193ed31339f0d to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/a8fc006fc607439e884193ed31339f0d 2023-03-29 03:54:40,793 DEBUG [StoreCloser-TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/310be1c32da64f1295d33cad9bb2c787 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/310be1c32da64f1295d33cad9bb2c787 2023-03-29 03:54:40,794 DEBUG [StoreCloser-TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/2df21fa89e2847fdaa6e5c1978c4cf1e to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/2df21fa89e2847fdaa6e5c1978c4cf1e 2023-03-29 03:54:40,795 DEBUG [StoreCloser-TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/9c01fbdb1bda447ab51e87c017b39dd9 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/9c01fbdb1bda447ab51e87c017b39dd9 2023-03-29 03:54:40,796 DEBUG [StoreCloser-TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/9318058e9cb948d5b9aa3fd907a7ec82 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/9318058e9cb948d5b9aa3fd907a7ec82 2023-03-29 03:54:40,797 DEBUG [StoreCloser-TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/da2a464998bf47e9a6082c65e09a7fa1 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/da2a464998bf47e9a6082c65e09a7fa1 2023-03-29 03:54:40,797 DEBUG [StoreCloser-TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/9beaaea60d37429d8490eeb284323256 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/9beaaea60d37429d8490eeb284323256 2023-03-29 03:54:40,798 DEBUG [StoreCloser-TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/e9f0af882d4846c99046a67a1818729b to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/e9f0af882d4846c99046a67a1818729b 2023-03-29 03:54:40,799 DEBUG [StoreCloser-TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/ae356b7a53184a01ba044c066a5c5acb to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/ae356b7a53184a01ba044c066a5c5acb 2023-03-29 03:54:40,800 DEBUG [StoreCloser-TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/2bd37e8188f44cc49ca55220c82a7025 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/2bd37e8188f44cc49ca55220c82a7025 2023-03-29 03:54:40,801 DEBUG [StoreCloser-TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/fda0e2948cf147e9a1efaa7733436dad to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/fda0e2948cf147e9a1efaa7733436dad 2023-03-29 03:54:40,802 DEBUG [StoreCloser-TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/3f0ed081b874469e8224777187f9d70a to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/3f0ed081b874469e8224777187f9d70a 2023-03-29 03:54:40,803 DEBUG [StoreCloser-TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/0d5a970b176e497dbf41a2abbce4c4d4 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/0d5a970b176e497dbf41a2abbce4c4d4 2023-03-29 03:54:40,804 DEBUG [StoreCloser-TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/7ecb6ce9ba7941ba80c3bed0abe1079d to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/7ecb6ce9ba7941ba80c3bed0abe1079d 2023-03-29 03:54:40,806 DEBUG [StoreCloser-TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/3110413ad6634b3e8578f2e900123436 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/3110413ad6634b3e8578f2e900123436 2023-03-29 03:54:40,807 DEBUG [StoreCloser-TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/85008277797347258066bb667b5c6765 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/85008277797347258066bb667b5c6765 2023-03-29 03:54:40,808 DEBUG [StoreCloser-TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/5d91b581dce24de0a3dda2eb48ebc725 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/5d91b581dce24de0a3dda2eb48ebc725 2023-03-29 03:54:40,809 DEBUG [StoreCloser-TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/f4c67b9d9ba04dc7bf95c7e502fbf7b8 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/f4c67b9d9ba04dc7bf95c7e502fbf7b8 2023-03-29 03:54:40,811 DEBUG [StoreCloser-TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/84d423adfc6d4ffb98ddc2e54747c29d to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/84d423adfc6d4ffb98ddc2e54747c29d 2023-03-29 03:54:40,812 DEBUG [StoreCloser-TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/48a0bdf086464a46b94c6707ecb26a38 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/48a0bdf086464a46b94c6707ecb26a38 2023-03-29 03:54:40,813 DEBUG [StoreCloser-TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/7b25a644464b4ea8b5d49653d4f65ed8 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/7b25a644464b4ea8b5d49653d4f65ed8 2023-03-29 03:54:40,813 DEBUG [StoreCloser-TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/d9cb6508c43d48328ac893ce7b94a47b to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/d9cb6508c43d48328ac893ce7b94a47b 2023-03-29 03:54:40,814 DEBUG [StoreCloser-TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/095f42e40c1b4d29bd6ad059f691b7ba to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/095f42e40c1b4d29bd6ad059f691b7ba 2023-03-29 03:54:40,815 DEBUG [StoreCloser-TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/36d6fe4241854e6183136f1efaaf5468 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/36d6fe4241854e6183136f1efaaf5468 2023-03-29 03:54:40,816 DEBUG [StoreCloser-TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/03870896a22c467ca5b5c53f7b8dfca7 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/03870896a22c467ca5b5c53f7b8dfca7 2023-03-29 03:54:40,817 DEBUG [StoreCloser-TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/7889044ff9464ebcbfecf8c5e2d96c9d to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/7889044ff9464ebcbfecf8c5e2d96c9d 2023-03-29 03:54:40,818 DEBUG [StoreCloser-TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/ee989dac96ba435993730a3b8609b30e to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/ee989dac96ba435993730a3b8609b30e 2023-03-29 03:54:40,819 DEBUG [StoreCloser-TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/840458d7ead84332b813b26368cf6098 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/840458d7ead84332b813b26368cf6098 2023-03-29 03:54:40,820 DEBUG [StoreCloser-TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/3c65c3c9e46b462c9a4702df8ae462e2 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/3c65c3c9e46b462c9a4702df8ae462e2 2023-03-29 03:54:40,821 DEBUG [StoreCloser-TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/4c7ab22bf037435cac07e549ca98ed26 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/4c7ab22bf037435cac07e549ca98ed26 2023-03-29 03:54:40,824 DEBUG [StoreCloser-TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/2c0c3cd702474babb481d56d8248c279, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/1b91a4cdcb154fb3ad48aa9f4dcd49c0, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/7c05e2107b844ff483fc71526dcdf5fc, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/7650c3e49e204614842a42aaff888b4c, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/d948c26f8cd54373b4ec896ca7b375ef, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/cb28c49a353e49f89ee3ad13ce3fc034, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/92b9a8817d904fa19b94c22ae40ccc44, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/de48ee89f80a4f859f6c82f6e70bb8ae, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/4b8e00de4e434030abb2a3de00c377ab, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/e61e6c7532674f6298e462f6f02afc5c, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/71bcf77814a14be49fba329ee74aefb1, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/6e49a08930004a6096945b85cb0c9b78, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/e7627510cac2403ab1578315b8034ee5, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/eb60a436a7fc4f439a5c78a68210c89f, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/20a663170f70472599a8d4c8e072398a, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/0bd911f13f094a62bbedd35e6faf71a0, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/9ec3cf0242d448aabca6081ed30f18ee, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/f225a9c95c3d4fc48bb6cca927769502, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/42ebf4470f2d4934a91ed79736a69e64, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/598c8fa333a14f6487ed9d8cdcf695ff, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/10c77866d5bb48428ea73938624f67cf, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/6bac5dcfc5c641f3a51c26d57357673c, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/d85df311c0f349299d2007f6b7007c13, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/8ca887db35ab496d86c542caaf1381dd, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/edb0848871f547e795a3ac5c2940f746, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/482e057972164ba088f55f944e170dac, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/b8f06cbe19994d249946f31238e303a6, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/1b7af5cd63364cabb3e0978af7b359c4, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/027c411e7f2a4d1fa6901eb675441ca1, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/4e0cff8b04d544758103e72d560e3630] to archive 2023-03-29 03:54:40,825 DEBUG [StoreCloser-TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-29 03:54:40,827 DEBUG [StoreCloser-TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/2c0c3cd702474babb481d56d8248c279 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/2c0c3cd702474babb481d56d8248c279 2023-03-29 03:54:40,828 DEBUG [StoreCloser-TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/1b91a4cdcb154fb3ad48aa9f4dcd49c0 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/1b91a4cdcb154fb3ad48aa9f4dcd49c0 2023-03-29 03:54:40,829 DEBUG [StoreCloser-TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/7c05e2107b844ff483fc71526dcdf5fc to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/7c05e2107b844ff483fc71526dcdf5fc 2023-03-29 03:54:40,831 DEBUG [StoreCloser-TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/7650c3e49e204614842a42aaff888b4c to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/7650c3e49e204614842a42aaff888b4c 2023-03-29 03:54:40,832 DEBUG [StoreCloser-TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/d948c26f8cd54373b4ec896ca7b375ef to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/d948c26f8cd54373b4ec896ca7b375ef 2023-03-29 03:54:40,833 DEBUG [StoreCloser-TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/cb28c49a353e49f89ee3ad13ce3fc034 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/cb28c49a353e49f89ee3ad13ce3fc034 2023-03-29 03:54:40,834 DEBUG [StoreCloser-TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/92b9a8817d904fa19b94c22ae40ccc44 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/92b9a8817d904fa19b94c22ae40ccc44 2023-03-29 03:54:40,836 DEBUG [StoreCloser-TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/de48ee89f80a4f859f6c82f6e70bb8ae to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/de48ee89f80a4f859f6c82f6e70bb8ae 2023-03-29 03:54:40,837 DEBUG [StoreCloser-TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/4b8e00de4e434030abb2a3de00c377ab to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/4b8e00de4e434030abb2a3de00c377ab 2023-03-29 03:54:40,838 DEBUG [StoreCloser-TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/e61e6c7532674f6298e462f6f02afc5c to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/e61e6c7532674f6298e462f6f02afc5c 2023-03-29 03:54:40,839 DEBUG [StoreCloser-TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/71bcf77814a14be49fba329ee74aefb1 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/71bcf77814a14be49fba329ee74aefb1 2023-03-29 03:54:40,840 DEBUG [StoreCloser-TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/6e49a08930004a6096945b85cb0c9b78 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/6e49a08930004a6096945b85cb0c9b78 2023-03-29 03:54:40,841 DEBUG [StoreCloser-TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/e7627510cac2403ab1578315b8034ee5 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/e7627510cac2403ab1578315b8034ee5 2023-03-29 03:54:40,842 DEBUG [StoreCloser-TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/eb60a436a7fc4f439a5c78a68210c89f to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/eb60a436a7fc4f439a5c78a68210c89f 2023-03-29 03:54:40,844 DEBUG [StoreCloser-TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/20a663170f70472599a8d4c8e072398a to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/20a663170f70472599a8d4c8e072398a 2023-03-29 03:54:40,845 DEBUG [StoreCloser-TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/0bd911f13f094a62bbedd35e6faf71a0 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/0bd911f13f094a62bbedd35e6faf71a0 2023-03-29 03:54:40,846 DEBUG [StoreCloser-TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/9ec3cf0242d448aabca6081ed30f18ee to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/9ec3cf0242d448aabca6081ed30f18ee 2023-03-29 03:54:40,847 DEBUG [StoreCloser-TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/f225a9c95c3d4fc48bb6cca927769502 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/f225a9c95c3d4fc48bb6cca927769502 2023-03-29 03:54:40,848 DEBUG [StoreCloser-TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/42ebf4470f2d4934a91ed79736a69e64 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/42ebf4470f2d4934a91ed79736a69e64 2023-03-29 03:54:40,849 DEBUG [StoreCloser-TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/598c8fa333a14f6487ed9d8cdcf695ff to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/598c8fa333a14f6487ed9d8cdcf695ff 2023-03-29 03:54:40,851 DEBUG [StoreCloser-TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/10c77866d5bb48428ea73938624f67cf to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/10c77866d5bb48428ea73938624f67cf 2023-03-29 03:54:40,852 DEBUG [StoreCloser-TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/6bac5dcfc5c641f3a51c26d57357673c to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/6bac5dcfc5c641f3a51c26d57357673c 2023-03-29 03:54:40,853 DEBUG [StoreCloser-TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/d85df311c0f349299d2007f6b7007c13 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/d85df311c0f349299d2007f6b7007c13 2023-03-29 03:54:40,854 DEBUG [StoreCloser-TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/8ca887db35ab496d86c542caaf1381dd to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/8ca887db35ab496d86c542caaf1381dd 2023-03-29 03:54:40,855 DEBUG [StoreCloser-TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/edb0848871f547e795a3ac5c2940f746 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/edb0848871f547e795a3ac5c2940f746 2023-03-29 03:54:40,856 DEBUG [StoreCloser-TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/482e057972164ba088f55f944e170dac to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/482e057972164ba088f55f944e170dac 2023-03-29 03:54:40,857 DEBUG [StoreCloser-TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/b8f06cbe19994d249946f31238e303a6 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/b8f06cbe19994d249946f31238e303a6 2023-03-29 03:54:40,858 DEBUG [StoreCloser-TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/1b7af5cd63364cabb3e0978af7b359c4 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/1b7af5cd63364cabb3e0978af7b359c4 2023-03-29 03:54:40,859 DEBUG [StoreCloser-TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/027c411e7f2a4d1fa6901eb675441ca1 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/027c411e7f2a4d1fa6901eb675441ca1 2023-03-29 03:54:40,860 DEBUG [StoreCloser-TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/4e0cff8b04d544758103e72d560e3630 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/4e0cff8b04d544758103e72d560e3630 2023-03-29 03:54:40,863 DEBUG [StoreCloser-TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/db71cfdcf2df4477b59fafaae3b40be2, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/c13f379d2c624f74bce0bc02627e55e0, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/ab983ba6a9f2490bb7cb93a11d31aaae, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/5d659d756248488f8265d3fab5c18249, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/316e75c9b5c5445c925de3c23b28ae5a, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/2939f52325594fd1b186471ff6fdc6b1, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/d1741c9d403a4407826e040016253bb3, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/0cd9cf0676354fc785b2c47b7596767f, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/a498ba292d1e40d6b0e1bc8086984e2d, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/176d6b10be3540cfa42110576e255a3d, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/445f38b2f5ed44feb05ef4041381382f, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/49676f4e28d94dd69ac114f618221c21, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/5a497fa220894f30bf9513d7734546c7, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/26be1f3657ce4301bc411a7166777796, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/362fbe53edaf44c8acf6a621729edb1f, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/679d6f71cb274c37b93dcd22df4480ad, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/c5c070e21e6c4a108512e2e4dc30a536, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/95627fb6b16f43d6af9ba010989d502c, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/fc31bd2e87874274a4ae87be71c1fda6, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/e0ed94b94124416ea2405f2b4321b442, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/ec55b793b5674e3ba492c683b3e3825e, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/a3a629d6a981463c912ac67e9b5bad97, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/18d80ceef9ac42d19f94a1e2460ce51a, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/9da409063e004b1fa5f88ca35e8f6fcb, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/9911eba33ab64624aae71a78afee6e15, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/418786d53267494da51baa6c756647dd, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/b093b0efc8f545c6bf91b8cd62d78d2a, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/3599483f9cad42e4b5c15bcc3f90b48a, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/3a48a2d6b83f4b79b4e185d95cc816f4] to archive 2023-03-29 03:54:40,864 DEBUG [StoreCloser-TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-29 03:54:40,866 DEBUG [StoreCloser-TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/db71cfdcf2df4477b59fafaae3b40be2 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/db71cfdcf2df4477b59fafaae3b40be2 2023-03-29 03:54:40,867 DEBUG [StoreCloser-TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/c13f379d2c624f74bce0bc02627e55e0 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/c13f379d2c624f74bce0bc02627e55e0 2023-03-29 03:54:40,868 DEBUG [StoreCloser-TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/ab983ba6a9f2490bb7cb93a11d31aaae to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/ab983ba6a9f2490bb7cb93a11d31aaae 2023-03-29 03:54:40,868 DEBUG [StoreCloser-TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/5d659d756248488f8265d3fab5c18249 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/5d659d756248488f8265d3fab5c18249 2023-03-29 03:54:40,869 DEBUG [StoreCloser-TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/316e75c9b5c5445c925de3c23b28ae5a to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/316e75c9b5c5445c925de3c23b28ae5a 2023-03-29 03:54:40,871 DEBUG [StoreCloser-TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/2939f52325594fd1b186471ff6fdc6b1 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/2939f52325594fd1b186471ff6fdc6b1 2023-03-29 03:54:40,872 DEBUG [StoreCloser-TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/d1741c9d403a4407826e040016253bb3 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/d1741c9d403a4407826e040016253bb3 2023-03-29 03:54:40,873 DEBUG [StoreCloser-TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/0cd9cf0676354fc785b2c47b7596767f to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/0cd9cf0676354fc785b2c47b7596767f 2023-03-29 03:54:40,874 DEBUG [StoreCloser-TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/a498ba292d1e40d6b0e1bc8086984e2d to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/a498ba292d1e40d6b0e1bc8086984e2d 2023-03-29 03:54:40,875 DEBUG [StoreCloser-TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/176d6b10be3540cfa42110576e255a3d to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/176d6b10be3540cfa42110576e255a3d 2023-03-29 03:54:40,876 DEBUG [StoreCloser-TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/445f38b2f5ed44feb05ef4041381382f to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/445f38b2f5ed44feb05ef4041381382f 2023-03-29 03:54:40,877 DEBUG [StoreCloser-TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/49676f4e28d94dd69ac114f618221c21 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/49676f4e28d94dd69ac114f618221c21 2023-03-29 03:54:40,878 DEBUG [StoreCloser-TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/5a497fa220894f30bf9513d7734546c7 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/5a497fa220894f30bf9513d7734546c7 2023-03-29 03:54:40,879 DEBUG [StoreCloser-TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/26be1f3657ce4301bc411a7166777796 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/26be1f3657ce4301bc411a7166777796 2023-03-29 03:54:40,880 DEBUG [StoreCloser-TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/362fbe53edaf44c8acf6a621729edb1f to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/362fbe53edaf44c8acf6a621729edb1f 2023-03-29 03:54:40,881 DEBUG [StoreCloser-TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/679d6f71cb274c37b93dcd22df4480ad to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/679d6f71cb274c37b93dcd22df4480ad 2023-03-29 03:54:40,882 DEBUG [StoreCloser-TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/c5c070e21e6c4a108512e2e4dc30a536 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/c5c070e21e6c4a108512e2e4dc30a536 2023-03-29 03:54:40,883 DEBUG [StoreCloser-TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/95627fb6b16f43d6af9ba010989d502c to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/95627fb6b16f43d6af9ba010989d502c 2023-03-29 03:54:40,884 DEBUG [StoreCloser-TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/fc31bd2e87874274a4ae87be71c1fda6 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/fc31bd2e87874274a4ae87be71c1fda6 2023-03-29 03:54:40,885 DEBUG [StoreCloser-TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/e0ed94b94124416ea2405f2b4321b442 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/e0ed94b94124416ea2405f2b4321b442 2023-03-29 03:54:40,886 DEBUG [StoreCloser-TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/ec55b793b5674e3ba492c683b3e3825e to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/ec55b793b5674e3ba492c683b3e3825e 2023-03-29 03:54:40,887 DEBUG [StoreCloser-TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/a3a629d6a981463c912ac67e9b5bad97 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/a3a629d6a981463c912ac67e9b5bad97 2023-03-29 03:54:40,888 DEBUG [StoreCloser-TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/18d80ceef9ac42d19f94a1e2460ce51a to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/18d80ceef9ac42d19f94a1e2460ce51a 2023-03-29 03:54:40,889 DEBUG [StoreCloser-TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/9da409063e004b1fa5f88ca35e8f6fcb to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/9da409063e004b1fa5f88ca35e8f6fcb 2023-03-29 03:54:40,890 DEBUG [StoreCloser-TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/9911eba33ab64624aae71a78afee6e15 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/9911eba33ab64624aae71a78afee6e15 2023-03-29 03:54:40,891 DEBUG [StoreCloser-TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/418786d53267494da51baa6c756647dd to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/418786d53267494da51baa6c756647dd 2023-03-29 03:54:40,892 DEBUG [StoreCloser-TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/b093b0efc8f545c6bf91b8cd62d78d2a to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/b093b0efc8f545c6bf91b8cd62d78d2a 2023-03-29 03:54:40,893 DEBUG [StoreCloser-TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/3599483f9cad42e4b5c15bcc3f90b48a to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/3599483f9cad42e4b5c15bcc3f90b48a 2023-03-29 03:54:40,894 DEBUG [StoreCloser-TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/3a48a2d6b83f4b79b4e185d95cc816f4 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/3a48a2d6b83f4b79b4e185d95cc816f4 2023-03-29 03:54:40,898 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/recovered.edits/579.seqid, newMaxSeqId=579, maxSeqId=4 2023-03-29 03:54:40,899 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0. 2023-03-29 03:54:40,899 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for ee0c8ff38ef29b37b45871dfc89897f0: 2023-03-29 03:54:40,900 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed ee0c8ff38ef29b37b45871dfc89897f0 2023-03-29 03:54:40,901 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=ee0c8ff38ef29b37b45871dfc89897f0, regionState=CLOSED 2023-03-29 03:54:40,901 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680062080901"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680062080901"}]},"ts":"1680062080901"} 2023-03-29 03:54:40,903 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=26, resume processing ppid=25 2023-03-29 03:54:40,903 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=26, ppid=25, state=SUCCESS; CloseRegionProcedure ee0c8ff38ef29b37b45871dfc89897f0, server=jenkins-hbase4.apache.org,35599,1680062018534 in 1.9350 sec 2023-03-29 03:54:40,904 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=25, resume processing ppid=24 2023-03-29 03:54:40,905 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=25, ppid=24, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=ee0c8ff38ef29b37b45871dfc89897f0, UNASSIGN in 1.9400 sec 2023-03-29 03:54:40,905 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680062080905"}]},"ts":"1680062080905"} 2023-03-29 03:54:40,906 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-03-29 03:54:40,908 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-03-29 03:54:40,909 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=24, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 2.0470 sec 2023-03-29 03:54:40,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-03-29 03:54:40,969 INFO [Listener at localhost/35027] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 24 completed 2023-03-29 03:54:40,970 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-03-29 03:54:40,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure2.ProcedureExecutor(1029): Stored pid=27, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-03-29 03:54:40,972 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=27, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-29 03:54:40,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=27 2023-03-29 03:54:40,972 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=27, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-29 03:54:40,977 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/.tmp/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0 2023-03-29 03:54:40,979 DEBUG [HFileArchiver-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/.tmp/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A, FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/.tmp/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B, FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/.tmp/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C, FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/.tmp/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/recovered.edits] 2023-03-29 03:54:40,981 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/.tmp/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/566d7c29b94d4ebb83418981213e736b to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/566d7c29b94d4ebb83418981213e736b 2023-03-29 03:54:40,982 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/.tmp/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/9ca46251e46a469a9ce35c295ce6ff8c to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/A/9ca46251e46a469a9ce35c295ce6ff8c 2023-03-29 03:54:40,984 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/.tmp/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/8e005e41e7574d39a03fb3c055e72646 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/8e005e41e7574d39a03fb3c055e72646 2023-03-29 03:54:40,985 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/.tmp/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/c7d173249d28446a8a0c44ef3b307438 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/B/c7d173249d28446a8a0c44ef3b307438 2023-03-29 03:54:40,988 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/.tmp/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/5fe2a8cf6e754b26a81a0e54d35e9262 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/5fe2a8cf6e754b26a81a0e54d35e9262 2023-03-29 03:54:40,989 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/.tmp/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/e0c5286f16de44849db233c5ba280e34 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/C/e0c5286f16de44849db233c5ba280e34 2023-03-29 03:54:40,992 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/.tmp/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/recovered.edits/579.seqid to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0/recovered.edits/579.seqid 2023-03-29 03:54:40,992 DEBUG [HFileArchiver-5] backup.HFileArchiver(596): Deleted hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/.tmp/data/default/TestAcidGuarantees/ee0c8ff38ef29b37b45871dfc89897f0 2023-03-29 03:54:40,993 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-29 03:54:40,993 DEBUG [PEWorker-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-03-29 03:54:40,994 DEBUG [PEWorker-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-03-29 03:54:40,997 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303290951514e9bff4e8b883f3177143217c3 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303290951514e9bff4e8b883f3177143217c3 2023-03-29 03:54:40,998 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303291c91e59956264bb196bffe9a54668f0f to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303291c91e59956264bb196bffe9a54668f0f 2023-03-29 03:54:40,999 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303292ac6c3e5a4e34171b89245b9d885f590 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303292ac6c3e5a4e34171b89245b9d885f590 2023-03-29 03:54:41,000 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303292d4d17beae214d899c9c1e28710823d0 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303292d4d17beae214d899c9c1e28710823d0 2023-03-29 03:54:41,001 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303293160144868894cae9f5adc7b59c0f97a to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303293160144868894cae9f5adc7b59c0f97a 2023-03-29 03:54:41,002 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032933047437159840588e1b67a8cca0d4f4 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032933047437159840588e1b67a8cca0d4f4 2023-03-29 03:54:41,003 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303294b2d07f1d98844828806fee2b40fce57 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303294b2d07f1d98844828806fee2b40fce57 2023-03-29 03:54:41,004 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303294bbf348ddcd9415fbcbb5264b4ceeb1f to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303294bbf348ddcd9415fbcbb5264b4ceeb1f 2023-03-29 03:54:41,006 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303294e2b42a3127a4425b8b5bf7331b68b77 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303294e2b42a3127a4425b8b5bf7331b68b77 2023-03-29 03:54:41,007 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032951f58a685fe040ee839f6787143dc1a0 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032951f58a685fe040ee839f6787143dc1a0 2023-03-29 03:54:41,008 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032979ca7dabb8224fa2aeb1239cff580b28 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032979ca7dabb8224fa2aeb1239cff580b28 2023-03-29 03:54:41,009 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303297b553edd067e4435bee15980d84e7c2b to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303297b553edd067e4435bee15980d84e7c2b 2023-03-29 03:54:41,010 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032989e1b96c544e41ddaf1774bdc941750f to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032989e1b96c544e41ddaf1774bdc941750f 2023-03-29 03:54:41,011 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032990bc2d9172534572a80d2c43dd21950e to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032990bc2d9172534572a80d2c43dd21950e 2023-03-29 03:54:41,013 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303299a24b2a7a4244516a6426c9b489534b5 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303299a24b2a7a4244516a6426c9b489534b5 2023-03-29 03:54:41,014 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303299cfc87fe4d244599a37c9d969df565fe to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303299cfc87fe4d244599a37c9d969df565fe 2023-03-29 03:54:41,015 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230329acb2a65e0a5e42e5814e41e602bbf68b to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230329acb2a65e0a5e42e5814e41e602bbf68b 2023-03-29 03:54:41,016 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230329af83388d530e44c28e89f9f53a52ccec to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230329af83388d530e44c28e89f9f53a52ccec 2023-03-29 03:54:41,018 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230329b1264c03ece04817bee46fb092e80be3 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230329b1264c03ece04817bee46fb092e80be3 2023-03-29 03:54:41,019 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230329cd337a000bbe48d4a04704fc0cae1167 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230329cd337a000bbe48d4a04704fc0cae1167 2023-03-29 03:54:41,020 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230329cd7affa96cb3456b9fea1ebe36999263 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230329cd7affa96cb3456b9fea1ebe36999263 2023-03-29 03:54:41,022 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230329d197d0f0bf9143c78a392ba1efcef176 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230329d197d0f0bf9143c78a392ba1efcef176 2023-03-29 03:54:41,023 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230329d6f3a9ea320b4743b498521ff3f428e3 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230329d6f3a9ea320b4743b498521ff3f428e3 2023-03-29 03:54:41,024 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230329dbd47d51f45540ac9b38ac80e08db86a to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230329dbd47d51f45540ac9b38ac80e08db86a 2023-03-29 03:54:41,025 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230329dee91d61916c44d9ad0d27e0cf1e8965 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230329dee91d61916c44d9ad0d27e0cf1e8965 2023-03-29 03:54:41,026 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230329e48d090c005f40b68d320494d16ccbcf to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230329e48d090c005f40b68d320494d16ccbcf 2023-03-29 03:54:41,027 DEBUG [PEWorker-3] backup.HFileArchiver(596): Deleted hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-03-29 03:54:41,029 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=27, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-29 03:54:41,031 WARN [PEWorker-3] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-03-29 03:54:41,033 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-03-29 03:54:41,033 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=27, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-29 03:54:41,034 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-03-29 03:54:41,034 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1680062081034"}]},"ts":"9223372036854775807"} 2023-03-29 03:54:41,035 INFO [PEWorker-3] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-03-29 03:54:41,035 DEBUG [PEWorker-3] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => ee0c8ff38ef29b37b45871dfc89897f0, NAME => 'TestAcidGuarantees,,1680062047208.ee0c8ff38ef29b37b45871dfc89897f0.', STARTKEY => '', ENDKEY => ''}] 2023-03-29 03:54:41,035 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-03-29 03:54:41,035 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1680062081035"}]},"ts":"9223372036854775807"} 2023-03-29 03:54:41,037 INFO [PEWorker-3] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-03-29 03:54:41,039 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(130): Finished pid=27, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-29 03:54:41,040 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=27, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 69 msec 2023-03-29 03:54:41,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=27 2023-03-29 03:54:41,073 INFO [Listener at localhost/35027] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 27 completed 2023-03-29 03:54:41,085 INFO [Listener at localhost/35027] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobMixedAtomicity Thread=226 (was 236), OpenFileDescriptor=512 (was 576), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=487 (was 528), ProcessCount=169 (was 170), AvailableMemoryMB=6835 (was 7074) 2023-03-29 03:54:41,096 INFO [Listener at localhost/35027] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testGetAtomicity Thread=226, OpenFileDescriptor=512, MaxFileDescriptor=60000, SystemLoadAverage=487, ProcessCount=169, AvailableMemoryMB=6834 2023-03-29 03:54:41,098 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-29 03:54:41,098 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-29 03:54:41,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure2.ProcedureExecutor(1029): Stored pid=28, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-03-29 03:54:41,100 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-03-29 03:54:41,101 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 28 2023-03-29 03:54:41,101 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-03-29 03:54:41,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-03-29 03:54:41,102 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/.tmp/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980 2023-03-29 03:54:41,103 DEBUG [HFileArchiver-6] backup.HFileArchiver(153): Directory hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/.tmp/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980 empty. 2023-03-29 03:54:41,103 DEBUG [HFileArchiver-6] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/.tmp/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980 2023-03-29 03:54:41,103 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-29 03:54:41,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-03-29 03:54:41,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-03-29 03:54:41,511 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-29 03:54:41,513 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 2ed55fa255954e1310f6b9fd96e28980, NAME => 'TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/.tmp 2023-03-29 03:54:41,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-03-29 03:54:41,919 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-29 03:54:41,919 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 2ed55fa255954e1310f6b9fd96e28980, disabling compactions & flushes 2023-03-29 03:54:41,919 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:54:41,919 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:54:41,919 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. after waiting 0 ms 2023-03-29 03:54:41,919 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:54:41,919 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:54:41,919 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 2ed55fa255954e1310f6b9fd96e28980: 2023-03-29 03:54:41,921 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-03-29 03:54:41,922 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680062081922"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680062081922"}]},"ts":"1680062081922"} 2023-03-29 03:54:41,923 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-03-29 03:54:41,924 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-03-29 03:54:41,924 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680062081924"}]},"ts":"1680062081924"} 2023-03-29 03:54:41,925 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-03-29 03:54:41,928 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=2ed55fa255954e1310f6b9fd96e28980, ASSIGN}] 2023-03-29 03:54:41,929 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=2ed55fa255954e1310f6b9fd96e28980, ASSIGN 2023-03-29 03:54:41,930 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=2ed55fa255954e1310f6b9fd96e28980, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,35599,1680062018534; forceNewPlan=false, retain=false 2023-03-29 03:54:42,081 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=2ed55fa255954e1310f6b9fd96e28980, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:42,081 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680062082081"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680062082081"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680062082081"}]},"ts":"1680062082081"} 2023-03-29 03:54:42,082 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=30, ppid=29, state=RUNNABLE; OpenRegionProcedure 2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534}] 2023-03-29 03:54:42,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-03-29 03:54:42,238 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:54:42,238 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 2ed55fa255954e1310f6b9fd96e28980, NAME => 'TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980.', STARTKEY => '', ENDKEY => ''} 2023-03-29 03:54:42,239 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 2ed55fa255954e1310f6b9fd96e28980 2023-03-29 03:54:42,239 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-29 03:54:42,239 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 2ed55fa255954e1310f6b9fd96e28980 2023-03-29 03:54:42,239 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 2ed55fa255954e1310f6b9fd96e28980 2023-03-29 03:54:42,241 INFO [StoreOpener-2ed55fa255954e1310f6b9fd96e28980-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 2ed55fa255954e1310f6b9fd96e28980 2023-03-29 03:54:42,242 DEBUG [StoreOpener-2ed55fa255954e1310f6b9fd96e28980-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A 2023-03-29 03:54:42,243 DEBUG [StoreOpener-2ed55fa255954e1310f6b9fd96e28980-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A 2023-03-29 03:54:42,243 INFO [StoreOpener-2ed55fa255954e1310f6b9fd96e28980-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-29 03:54:42,243 INFO [StoreOpener-2ed55fa255954e1310f6b9fd96e28980-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 2ed55fa255954e1310f6b9fd96e28980 columnFamilyName A 2023-03-29 03:54:42,243 INFO [StoreOpener-2ed55fa255954e1310f6b9fd96e28980-1] regionserver.HStore(310): Store=2ed55fa255954e1310f6b9fd96e28980/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-29 03:54:42,244 INFO [StoreOpener-2ed55fa255954e1310f6b9fd96e28980-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 2ed55fa255954e1310f6b9fd96e28980 2023-03-29 03:54:42,245 DEBUG [StoreOpener-2ed55fa255954e1310f6b9fd96e28980-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B 2023-03-29 03:54:42,245 DEBUG [StoreOpener-2ed55fa255954e1310f6b9fd96e28980-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B 2023-03-29 03:54:42,245 INFO [StoreOpener-2ed55fa255954e1310f6b9fd96e28980-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-29 03:54:42,245 INFO [StoreOpener-2ed55fa255954e1310f6b9fd96e28980-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 2ed55fa255954e1310f6b9fd96e28980 columnFamilyName B 2023-03-29 03:54:42,246 INFO [StoreOpener-2ed55fa255954e1310f6b9fd96e28980-1] regionserver.HStore(310): Store=2ed55fa255954e1310f6b9fd96e28980/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-29 03:54:42,246 INFO [StoreOpener-2ed55fa255954e1310f6b9fd96e28980-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 2ed55fa255954e1310f6b9fd96e28980 2023-03-29 03:54:42,247 DEBUG [StoreOpener-2ed55fa255954e1310f6b9fd96e28980-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C 2023-03-29 03:54:42,247 DEBUG [StoreOpener-2ed55fa255954e1310f6b9fd96e28980-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C 2023-03-29 03:54:42,247 INFO [StoreOpener-2ed55fa255954e1310f6b9fd96e28980-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-29 03:54:42,247 INFO [StoreOpener-2ed55fa255954e1310f6b9fd96e28980-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 2ed55fa255954e1310f6b9fd96e28980 columnFamilyName C 2023-03-29 03:54:42,248 INFO [StoreOpener-2ed55fa255954e1310f6b9fd96e28980-1] regionserver.HStore(310): Store=2ed55fa255954e1310f6b9fd96e28980/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-29 03:54:42,248 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:54:42,248 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980 2023-03-29 03:54:42,249 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980 2023-03-29 03:54:42,250 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-29 03:54:42,251 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 2ed55fa255954e1310f6b9fd96e28980 2023-03-29 03:54:42,254 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-29 03:54:42,254 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 2ed55fa255954e1310f6b9fd96e28980; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63217749, jitterRate=-0.05798213183879852}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-29 03:54:42,254 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 2ed55fa255954e1310f6b9fd96e28980: 2023-03-29 03:54:42,255 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980., pid=30, masterSystemTime=1680062082233 2023-03-29 03:54:42,256 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:54:42,256 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:54:42,256 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=2ed55fa255954e1310f6b9fd96e28980, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:42,257 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680062082256"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1680062082256"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680062082256"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680062082256"}]},"ts":"1680062082256"} 2023-03-29 03:54:42,259 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=30, resume processing ppid=29 2023-03-29 03:54:42,259 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=30, ppid=29, state=SUCCESS; OpenRegionProcedure 2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 in 176 msec 2023-03-29 03:54:42,260 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=29, resume processing ppid=28 2023-03-29 03:54:42,260 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=29, ppid=28, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=2ed55fa255954e1310f6b9fd96e28980, ASSIGN in 331 msec 2023-03-29 03:54:42,261 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-03-29 03:54:42,261 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680062082261"}]},"ts":"1680062082261"} 2023-03-29 03:54:42,262 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-03-29 03:54:42,266 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-03-29 03:54:42,267 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=28, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1670 sec 2023-03-29 03:54:43,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-03-29 03:54:43,206 INFO [Listener at localhost/35027] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 28 completed 2023-03-29 03:54:43,207 DEBUG [Listener at localhost/35027] zookeeper.ReadOnlyZKClient(139): Connect 0x23c97092 to 127.0.0.1:52103 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-29 03:54:43,212 DEBUG [Listener at localhost/35027] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@70043cf0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-29 03:54:43,213 DEBUG [Listener at localhost/35027] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-29 03:54:43,214 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:37550, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-29 03:54:43,215 DEBUG [Listener at localhost/35027] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-03-29 03:54:43,217 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:37062, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-03-29 03:54:43,218 DEBUG [Listener at localhost/35027] zookeeper.ReadOnlyZKClient(139): Connect 0x5c124820 to 127.0.0.1:52103 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-29 03:54:43,222 DEBUG [Listener at localhost/35027] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@34b7fed0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-29 03:54:43,222 DEBUG [Listener at localhost/35027] zookeeper.ReadOnlyZKClient(139): Connect 0x3157d4df to 127.0.0.1:52103 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-29 03:54:43,226 DEBUG [Listener at localhost/35027] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7f4ad99d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-29 03:54:43,226 DEBUG [Listener at localhost/35027] zookeeper.ReadOnlyZKClient(139): Connect 0x3cf5bdb7 to 127.0.0.1:52103 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-29 03:54:43,231 DEBUG [Listener at localhost/35027] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@330b77da, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-29 03:54:43,231 DEBUG [Listener at localhost/35027] zookeeper.ReadOnlyZKClient(139): Connect 0x6919b2a0 to 127.0.0.1:52103 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-29 03:54:43,234 DEBUG [Listener at localhost/35027] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7810a231, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-29 03:54:43,235 DEBUG [Listener at localhost/35027] zookeeper.ReadOnlyZKClient(139): Connect 0x2ce5573e to 127.0.0.1:52103 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-29 03:54:43,239 DEBUG [Listener at localhost/35027] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@8cd7cb0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-29 03:54:43,239 DEBUG [Listener at localhost/35027] zookeeper.ReadOnlyZKClient(139): Connect 0x0ea1ca03 to 127.0.0.1:52103 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-29 03:54:43,244 DEBUG [Listener at localhost/35027] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6dea0f9b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-29 03:54:43,244 DEBUG [Listener at localhost/35027] zookeeper.ReadOnlyZKClient(139): Connect 0x433f251d to 127.0.0.1:52103 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-29 03:54:43,249 DEBUG [Listener at localhost/35027] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@434e0eab, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-29 03:54:43,250 DEBUG [Listener at localhost/35027] zookeeper.ReadOnlyZKClient(139): Connect 0x66ee0bd0 to 127.0.0.1:52103 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-29 03:54:43,253 DEBUG [Listener at localhost/35027] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5c43dddd, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-29 03:54:43,254 DEBUG [Listener at localhost/35027] zookeeper.ReadOnlyZKClient(139): Connect 0x3b4658f8 to 127.0.0.1:52103 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-29 03:54:43,258 DEBUG [Listener at localhost/35027] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2de064cb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-29 03:54:43,259 DEBUG [Listener at localhost/35027] zookeeper.ReadOnlyZKClient(139): Connect 0x5d2b992d to 127.0.0.1:52103 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-29 03:54:43,262 DEBUG [Listener at localhost/35027] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@54a39be8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-29 03:54:43,266 DEBUG [hconnection-0x3a3f66e8-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-29 03:54:43,266 DEBUG [hconnection-0x553dba4f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-29 03:54:43,267 DEBUG [hconnection-0x46e7c28a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-29 03:54:43,268 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:37562, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-29 03:54:43,268 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:37570, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-29 03:54:43,269 DEBUG [hconnection-0x382c66dc-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-29 03:54:43,269 DEBUG [hconnection-0x7041ef3a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-29 03:54:43,269 DEBUG [hconnection-0x24534ea6-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-29 03:54:43,269 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:54:43,269 DEBUG [hconnection-0x49bfcfd-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-29 03:54:43,269 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:37572, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-29 03:54:43,270 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:37610, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-29 03:54:43,272 DEBUG [hconnection-0x28b37c25-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-29 03:54:43,273 DEBUG [hconnection-0x25f7f993-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-29 03:54:43,273 DEBUG [hconnection-0x4b5ac466-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-29 03:54:43,275 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:37634, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-29 03:54:43,276 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:37600, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-29 03:54:43,277 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:37618, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-29 03:54:43,277 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:37586, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-29 03:54:43,278 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:37614, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-29 03:54:43,279 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:37646, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-29 03:54:43,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:54:43,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:54:43,284 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:54:43,284 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:43,284 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:54:43,284 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:54:43,284 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:43,284 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:43,286 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:43,287 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:43,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:43,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:43,287 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:43,287 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:54:43,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:43,287 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:43,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:54:43,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:43,288 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:54:43,289 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:54:43,289 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:54:43,289 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:54:43,289 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:43,289 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:54:43,290 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:54:43,290 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:54:43,290 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:54:43,290 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. started... 2023-03-29 03:54:43,291 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2ed55fa255954e1310f6b9fd96e28980 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-29 03:54:43,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on 2ed55fa255954e1310f6b9fd96e28980 2023-03-29 03:54:43,291 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. as already flushing 2023-03-29 03:54:43,291 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ed55fa255954e1310f6b9fd96e28980, store=A 2023-03-29 03:54:43,291 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:54:43,291 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ed55fa255954e1310f6b9fd96e28980, store=B 2023-03-29 03:54:43,291 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:54:43,291 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ed55fa255954e1310f6b9fd96e28980, store=C 2023-03-29 03:54:43,291 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:54:43,309 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:43,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37634 deadline: 1680062143305, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:43,311 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:43,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37610 deadline: 1680062143308, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:43,312 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:43,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37570 deadline: 1680062143309, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:43,313 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:43,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 5 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37614 deadline: 1680062143309, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:43,313 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:43,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37572 deadline: 1680062143310, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:43,313 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680062083301, max timestamp=1680062083308Num uniques 859; 2023-03-29 03:54:43,313 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-29 03:54:43,314 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680062083301, max timestamp=1680062083308Num uniques 859; 2023-03-29 03:54:43,315 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-29 03:54:43,313 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680062083301, max timestamp=1680062083308Num uniques 859; 2023-03-29 03:54:43,315 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-29 03:54:43,426 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:43,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37614 deadline: 1680062143422, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:43,426 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:43,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37572 deadline: 1680062143423, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:43,427 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:43,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37634 deadline: 1680062143423, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:43,427 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:43,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37610 deadline: 1680062143424, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:43,428 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:43,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37570 deadline: 1680062143424, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:43,528 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:43,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37614 deadline: 1680062143527, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:43,529 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:43,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37572 deadline: 1680062143527, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:43,529 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:43,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37634 deadline: 1680062143528, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:43,530 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:43,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37610 deadline: 1680062143528, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:43,530 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:43,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37570 deadline: 1680062143529, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:43,732 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:43,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37614 deadline: 1680062143730, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:43,733 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:43,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37572 deadline: 1680062143731, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:43,733 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:43,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37634 deadline: 1680062143731, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:43,733 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:43,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37570 deadline: 1680062143731, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:43,733 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:43,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37610 deadline: 1680062143732, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:43,736 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=13 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/A/d5416a3ab8c94b59b205a497e7ccf908 2023-03-29 03:54:43,760 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=13 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/B/35a0fd77f91045d795cab8be37f29b68 2023-03-29 03:54:44,036 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:44,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37614 deadline: 1680062144034, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:44,037 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:44,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37634 deadline: 1680062144035, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:44,038 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:44,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37572 deadline: 1680062144035, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:44,038 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:44,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37570 deadline: 1680062144036, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:44,038 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:44,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37610 deadline: 1680062144036, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:44,174 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=13 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/C/5a5183a24464403daf8eee49ef0652d3 2023-03-29 03:54:44,179 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/A/d5416a3ab8c94b59b205a497e7ccf908 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/d5416a3ab8c94b59b205a497e7ccf908 2023-03-29 03:54:44,184 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/d5416a3ab8c94b59b205a497e7ccf908, entries=150, sequenceid=13, filesize=11.6 K 2023-03-29 03:54:44,185 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/B/35a0fd77f91045d795cab8be37f29b68 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/35a0fd77f91045d795cab8be37f29b68 2023-03-29 03:54:44,189 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/35a0fd77f91045d795cab8be37f29b68, entries=150, sequenceid=13, filesize=11.6 K 2023-03-29 03:54:44,190 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/C/5a5183a24464403daf8eee49ef0652d3 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/5a5183a24464403daf8eee49ef0652d3 2023-03-29 03:54:44,194 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/5a5183a24464403daf8eee49ef0652d3, entries=150, sequenceid=13, filesize=11.6 K 2023-03-29 03:54:44,195 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=120.76 KB/123660 for 2ed55fa255954e1310f6b9fd96e28980 in 904ms, sequenceid=13, compaction requested=false 2023-03-29 03:54:44,195 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2ed55fa255954e1310f6b9fd96e28980: 2023-03-29 03:54:44,195 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:54:44,195 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:54:44,195 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:54:44,195 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:44,195 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:54:44,196 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:54:44,197 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:44,197 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:44,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:44,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:44,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:44,198 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:44,198 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:54:44,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:44,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:44,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:44,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:44,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:44,199 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:54:44,199 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@55c9c1ad[Count = 0] remaining members to acquire global barrier 2023-03-29 03:54:44,199 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:54:44,199 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:44,200 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:44,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:44,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:44,201 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:44,201 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:54:44,201 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:54:44,201 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:54:44,201 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:54:44,202 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:44,202 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:54:44,202 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:44,202 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:54:44,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:44,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:44,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:44,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:44,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:44,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:44,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:44,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:44,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:44,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:44,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:54:44,205 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:54:44,205 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:54:44,205 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:54:44,205 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:54:44,205 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:54:44,205 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:54:44,208 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:44,208 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:44,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:44,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:44,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:44,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:44,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:44,208 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:44,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:44,209 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:44,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:44,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:44,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:44,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:44,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:44,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:44,211 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:44,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:44,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:44,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:44,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:44,213 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:44,213 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:44,213 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:44,213 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:44,213 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:54:44,213 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:54:44,213 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:44,213 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:54:44,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:44,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:44,213 DEBUG [Thread-1741] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:54:44,213 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:44,213 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:44,214 DEBUG [Thread-1741] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:54:44,213 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:44,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:44,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:44,214 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:44,214 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:44,314 DEBUG [Thread-1741] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:54:44,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:54:44,316 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:54:44,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:54:44,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:54:44,317 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:54:44,317 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:44,317 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:54:44,317 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:54:44,318 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:44,318 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:44,321 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:44,321 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:44,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:44,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:44,328 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:44,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:44,328 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:54:44,328 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:44,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:54:44,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:44,329 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:54:44,329 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:54:44,329 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:54:44,329 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:54:44,329 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:44,329 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:54:44,330 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:54:44,330 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:54:44,330 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:54:44,330 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. started... 2023-03-29 03:54:44,330 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2ed55fa255954e1310f6b9fd96e28980 3/3 column families, dataSize=120.76 KB heapSize=345.14 KB 2023-03-29 03:54:44,330 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ed55fa255954e1310f6b9fd96e28980, store=A 2023-03-29 03:54:44,330 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-29 03:54:44,330 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ed55fa255954e1310f6b9fd96e28980, store=B 2023-03-29 03:54:44,330 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-29 03:54:44,330 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ed55fa255954e1310f6b9fd96e28980, store=C 2023-03-29 03:54:44,330 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-29 03:54:44,341 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=48 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/A/62af4785ca5e4f18b6bf0ebc2f1027fb 2023-03-29 03:54:44,353 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=48 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/B/08fbb1b29f924230963c70f920d33799 2023-03-29 03:54:44,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on 2ed55fa255954e1310f6b9fd96e28980 2023-03-29 03:54:44,540 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. as already flushing 2023-03-29 03:54:44,548 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:44,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37570 deadline: 1680062144544, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:44,548 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:44,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37610 deadline: 1680062144545, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:44,549 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:44,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37634 deadline: 1680062144546, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:44,549 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:44,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37572 deadline: 1680062144546, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:44,549 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:44,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37614 deadline: 1680062144547, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:44,651 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:44,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37610 deadline: 1680062144650, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:44,654 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:44,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37614 deadline: 1680062144651, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:44,654 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:44,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37570 deadline: 1680062144652, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:44,655 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:44,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37572 deadline: 1680062144652, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:44,655 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:44,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37634 deadline: 1680062144652, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:44,803 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=48 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/C/6e831407f49d477b94aa1daa8a1564ba 2023-03-29 03:54:44,818 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/A/62af4785ca5e4f18b6bf0ebc2f1027fb as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/62af4785ca5e4f18b6bf0ebc2f1027fb 2023-03-29 03:54:44,824 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/62af4785ca5e4f18b6bf0ebc2f1027fb, entries=150, sequenceid=48, filesize=11.6 K 2023-03-29 03:54:44,825 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/B/08fbb1b29f924230963c70f920d33799 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/08fbb1b29f924230963c70f920d33799 2023-03-29 03:54:44,829 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/08fbb1b29f924230963c70f920d33799, entries=150, sequenceid=48, filesize=11.6 K 2023-03-29 03:54:44,830 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/C/6e831407f49d477b94aa1daa8a1564ba as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/6e831407f49d477b94aa1daa8a1564ba 2023-03-29 03:54:44,835 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/6e831407f49d477b94aa1daa8a1564ba, entries=150, sequenceid=48, filesize=11.6 K 2023-03-29 03:54:44,835 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~345.09 KB/353376, currentSize=60.38 KB/61830 for 2ed55fa255954e1310f6b9fd96e28980 in 505ms, sequenceid=48, compaction requested=false 2023-03-29 03:54:44,836 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2ed55fa255954e1310f6b9fd96e28980: 2023-03-29 03:54:44,836 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:54:44,836 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:54:44,836 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:54:44,836 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:44,840 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:54:44,840 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:54:44,842 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:44,842 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:44,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:44,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:44,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:44,842 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:44,842 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:54:44,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:44,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:44,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:44,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:44,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:44,844 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:54:44,844 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1647bacc[Count = 0] remaining members to acquire global barrier 2023-03-29 03:54:44,844 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:54:44,844 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:44,846 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:44,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:44,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:44,846 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:54:44,846 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:54:44,846 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:54:44,846 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:44,846 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:54:44,847 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:44,847 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:54:44,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:44,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:44,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:44,847 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:44,848 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:54:44,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:44,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:44,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:44,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:44,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:44,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:44,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:44,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:54:44,850 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:54:44,850 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:54:44,850 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:54:44,850 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:54:44,850 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:54:44,850 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:54:44,852 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:44,852 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:44,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:44,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:44,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:44,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:44,852 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:44,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:44,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:44,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:44,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:44,853 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:44,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:44,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:44,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:44,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:44,854 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:44,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:44,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on 2ed55fa255954e1310f6b9fd96e28980 2023-03-29 03:54:44,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:44,854 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2ed55fa255954e1310f6b9fd96e28980 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-29 03:54:44,855 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ed55fa255954e1310f6b9fd96e28980, store=A 2023-03-29 03:54:44,855 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:54:44,855 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ed55fa255954e1310f6b9fd96e28980, store=B 2023-03-29 03:54:44,855 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:54:44,855 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ed55fa255954e1310f6b9fd96e28980, store=C 2023-03-29 03:54:44,855 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:54:44,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:44,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:44,859 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:44,859 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:44,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:44,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:44,860 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:44,860 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:44,860 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:44,860 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:44,860 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:44,860 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:44,860 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:44,860 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:44,860 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:54:44,860 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:54:44,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:44,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:44,860 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:54:44,860 DEBUG [Thread-1741] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:54:44,860 DEBUG [Thread-1741] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:54:44,877 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:44,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37614 deadline: 1680062144872, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:44,878 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:44,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37634 deadline: 1680062144873, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:44,878 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:44,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37610 deadline: 1680062144874, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:44,878 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:44,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37570 deadline: 1680062144874, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:44,880 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:44,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37572 deadline: 1680062144877, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:44,883 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=61 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/A/0f33d61018ff46b48f01ca6764fce7d1 2023-03-29 03:54:44,914 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=61 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/B/a5bb8db8363043c1907c8806bfff2078 2023-03-29 03:54:44,953 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=61 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/C/d629ddeaf72640d8bcc395035d9326d7 2023-03-29 03:54:44,961 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/A/0f33d61018ff46b48f01ca6764fce7d1 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/0f33d61018ff46b48f01ca6764fce7d1 2023-03-29 03:54:44,961 DEBUG [Thread-1741] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:54:44,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:54:44,964 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:54:44,966 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/0f33d61018ff46b48f01ca6764fce7d1, entries=150, sequenceid=61, filesize=11.6 K 2023-03-29 03:54:44,967 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/B/a5bb8db8363043c1907c8806bfff2078 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/a5bb8db8363043c1907c8806bfff2078 2023-03-29 03:54:44,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:54:44,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:54:44,968 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:54:44,968 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:44,968 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:54:44,968 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:54:44,969 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:44,969 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:44,970 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:44,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:44,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:44,971 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:44,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:44,971 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:44,972 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:44,972 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:54:44,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:54:44,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:44,972 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:54:44,973 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/a5bb8db8363043c1907c8806bfff2078, entries=150, sequenceid=61, filesize=11.6 K 2023-03-29 03:54:44,974 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:54:44,974 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:54:44,974 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:54:44,974 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:44,974 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:54:44,974 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:54:44,974 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:54:44,974 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:54:44,974 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. started... 2023-03-29 03:54:44,975 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. as already flushing 2023-03-29 03:54:44,975 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/C/d629ddeaf72640d8bcc395035d9326d7 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/d629ddeaf72640d8bcc395035d9326d7 2023-03-29 03:54:44,975 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:54:44,975 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:54:44,975 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:54:44,975 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:44,976 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:54:44,976 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:54:44,978 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:44,978 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:44,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:44,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:44,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:44,979 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:44,979 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:54:44,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:44,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:44,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:44,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:44,980 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:44,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37614 deadline: 1680062144978, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:44,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:44,981 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:44,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37570 deadline: 1680062144979, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:44,981 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:44,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37610 deadline: 1680062144979, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:44,981 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:44,981 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:54:44,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37634 deadline: 1680062144979, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:44,981 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:54:44,981 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:44,981 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@c9ff26b[Count = 0] remaining members to acquire global barrier 2023-03-29 03:54:44,983 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:44,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:44,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:44,983 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:54:44,983 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:54:44,983 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:54:44,983 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:44,983 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:54:44,984 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:44,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37572 deadline: 1680062144981, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:44,987 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:54:44,987 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:44,987 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:54:44,987 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:44,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:44,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:44,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:44,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:44,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:44,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:44,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:44,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:44,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:44,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:44,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:54:44,991 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:54:44,991 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:54:44,991 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:54:44,991 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:54:44,991 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:54:44,991 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:54:44,993 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/d629ddeaf72640d8bcc395035d9326d7, entries=150, sequenceid=61, filesize=11.6 K 2023-03-29 03:54:44,993 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:44,993 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:44,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:44,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:44,993 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:44,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:44,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:44,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:44,993 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 2ed55fa255954e1310f6b9fd96e28980 in 139ms, sequenceid=61, compaction requested=true 2023-03-29 03:54:44,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:44,994 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2ed55fa255954e1310f6b9fd96e28980: 2023-03-29 03:54:44,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:44,994 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:54:44,994 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-29 03:54:44,994 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-29 03:54:44,994 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:54:44,994 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:54:44,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:44,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:44,995 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:44,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:44,996 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-29 03:54:44,996 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1912): 2ed55fa255954e1310f6b9fd96e28980/A is initiating minor compaction (all files) 2023-03-29 03:54:44,996 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2ed55fa255954e1310f6b9fd96e28980/A in TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:54:44,996 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/d5416a3ab8c94b59b205a497e7ccf908, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/62af4785ca5e4f18b6bf0ebc2f1027fb, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/0f33d61018ff46b48f01ca6764fce7d1] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp, totalSize=34.8 K 2023-03-29 03:54:44,996 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-29 03:54:44,996 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1912): 2ed55fa255954e1310f6b9fd96e28980/B is initiating minor compaction (all files) 2023-03-29 03:54:44,996 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2ed55fa255954e1310f6b9fd96e28980/B in TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:54:44,997 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/35a0fd77f91045d795cab8be37f29b68, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/08fbb1b29f924230963c70f920d33799, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/a5bb8db8363043c1907c8806bfff2078] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp, totalSize=34.8 K 2023-03-29 03:54:44,997 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting d5416a3ab8c94b59b205a497e7ccf908, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1680062083282 2023-03-29 03:54:44,998 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 62af4785ca5e4f18b6bf0ebc2f1027fb, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1680062083421 2023-03-29 03:54:44,998 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 0f33d61018ff46b48f01ca6764fce7d1, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=61, earliestPutTs=1680062084540 2023-03-29 03:54:44,999 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 35a0fd77f91045d795cab8be37f29b68, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1680062083282 2023-03-29 03:54:44,999 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 08fbb1b29f924230963c70f920d33799, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1680062083421 2023-03-29 03:54:45,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:45,000 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting a5bb8db8363043c1907c8806bfff2078, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=61, earliestPutTs=1680062084540 2023-03-29 03:54:45,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:45,003 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:45,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:45,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:45,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:45,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:45,012 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:45,012 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:45,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:45,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:45,013 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:45,013 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:45,013 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:45,013 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:45,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:45,013 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:54:45,013 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:54:45,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:45,013 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:45,013 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:45,013 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] throttle.PressureAwareThroughputController(145): 2ed55fa255954e1310f6b9fd96e28980#B#compaction#229 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:54:45,013 DEBUG [Thread-1741] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:54:45,013 DEBUG [Thread-1741] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:54:45,013 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:45,014 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:45,021 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2ed55fa255954e1310f6b9fd96e28980#A#compaction#230 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:54:45,062 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/A/fb37541f10a54c0aa4f04af8de268e0c as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/fb37541f10a54c0aa4f04af8de268e0c 2023-03-29 03:54:45,068 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2ed55fa255954e1310f6b9fd96e28980/A of 2ed55fa255954e1310f6b9fd96e28980 into fb37541f10a54c0aa4f04af8de268e0c(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:54:45,068 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2ed55fa255954e1310f6b9fd96e28980: 2023-03-29 03:54:45,068 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980., storeName=2ed55fa255954e1310f6b9fd96e28980/A, priority=13, startTime=1680062084994; duration=0sec 2023-03-29 03:54:45,068 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:54:45,068 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-29 03:54:45,069 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-29 03:54:45,070 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1912): 2ed55fa255954e1310f6b9fd96e28980/C is initiating minor compaction (all files) 2023-03-29 03:54:45,070 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2ed55fa255954e1310f6b9fd96e28980/C in TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:54:45,070 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/5a5183a24464403daf8eee49ef0652d3, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/6e831407f49d477b94aa1daa8a1564ba, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/d629ddeaf72640d8bcc395035d9326d7] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp, totalSize=34.8 K 2023-03-29 03:54:45,070 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 5a5183a24464403daf8eee49ef0652d3, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1680062083282 2023-03-29 03:54:45,071 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 6e831407f49d477b94aa1daa8a1564ba, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1680062083421 2023-03-29 03:54:45,071 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting d629ddeaf72640d8bcc395035d9326d7, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=61, earliestPutTs=1680062084540 2023-03-29 03:54:45,076 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2ed55fa255954e1310f6b9fd96e28980#C#compaction#231 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:54:45,098 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/C/91ee23eb360842a29ac70105fa934f9f as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/91ee23eb360842a29ac70105fa934f9f 2023-03-29 03:54:45,104 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2ed55fa255954e1310f6b9fd96e28980/C of 2ed55fa255954e1310f6b9fd96e28980 into 91ee23eb360842a29ac70105fa934f9f(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:54:45,104 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2ed55fa255954e1310f6b9fd96e28980: 2023-03-29 03:54:45,104 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980., storeName=2ed55fa255954e1310f6b9fd96e28980/C, priority=13, startTime=1680062084994; duration=0sec 2023-03-29 03:54:45,104 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:54:45,113 DEBUG [Thread-1741] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:54:45,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:54:45,116 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:54:45,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:54:45,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:54:45,118 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:54:45,119 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:45,119 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:54:45,119 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:54:45,120 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:45,120 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:45,121 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:45,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:45,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:45,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:45,121 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:45,122 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:45,122 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:45,122 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:54:45,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:54:45,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:45,123 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:54:45,123 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:54:45,123 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:54:45,124 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:54:45,124 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:45,124 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:54:45,128 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:54:45,128 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:54:45,128 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:54:45,128 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. started... 2023-03-29 03:54:45,128 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2ed55fa255954e1310f6b9fd96e28980 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-29 03:54:45,129 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ed55fa255954e1310f6b9fd96e28980, store=A 2023-03-29 03:54:45,129 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:54:45,129 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ed55fa255954e1310f6b9fd96e28980, store=B 2023-03-29 03:54:45,129 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:54:45,129 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ed55fa255954e1310f6b9fd96e28980, store=C 2023-03-29 03:54:45,129 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:54:45,153 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=82 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/A/b884545047f64c17b7949a2c6b266711 2023-03-29 03:54:45,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on 2ed55fa255954e1310f6b9fd96e28980 2023-03-29 03:54:45,184 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. as already flushing 2023-03-29 03:54:45,228 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:45,228 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:45,228 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:45,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37570 deadline: 1680062145225, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:45,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37572 deadline: 1680062145225, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:45,229 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:45,229 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:45,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37634 deadline: 1680062145226, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:45,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37614 deadline: 1680062145225, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:45,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37610 deadline: 1680062145226, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:45,331 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:45,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37572 deadline: 1680062145330, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:45,332 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:45,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37570 deadline: 1680062145330, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:45,332 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:45,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37634 deadline: 1680062145330, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:45,332 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:45,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37610 deadline: 1680062145330, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:45,333 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:45,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37614 deadline: 1680062145330, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:45,460 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/B/2e93d88526c5484fbd124c2e5ffd8ed0 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/2e93d88526c5484fbd124c2e5ffd8ed0 2023-03-29 03:54:45,465 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2ed55fa255954e1310f6b9fd96e28980/B of 2ed55fa255954e1310f6b9fd96e28980 into 2e93d88526c5484fbd124c2e5ffd8ed0(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:54:45,465 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2ed55fa255954e1310f6b9fd96e28980: 2023-03-29 03:54:45,465 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980., storeName=2ed55fa255954e1310f6b9fd96e28980/B, priority=13, startTime=1680062084994; duration=0sec 2023-03-29 03:54:45,465 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:54:45,533 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:45,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37572 deadline: 1680062145533, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:45,535 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:45,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37570 deadline: 1680062145534, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:45,535 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:45,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37634 deadline: 1680062145534, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:45,536 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:45,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37614 deadline: 1680062145534, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:45,545 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:45,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37610 deadline: 1680062145544, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:45,579 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=82 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/B/37cbfd98fc4549a0b43ef46399fe115e 2023-03-29 03:54:45,591 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=82 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/C/ea1581cf7b08437dbfd19c053685f48d 2023-03-29 03:54:45,596 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/A/b884545047f64c17b7949a2c6b266711 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/b884545047f64c17b7949a2c6b266711 2023-03-29 03:54:45,600 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/b884545047f64c17b7949a2c6b266711, entries=150, sequenceid=82, filesize=11.6 K 2023-03-29 03:54:45,601 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/B/37cbfd98fc4549a0b43ef46399fe115e as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/37cbfd98fc4549a0b43ef46399fe115e 2023-03-29 03:54:45,606 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/37cbfd98fc4549a0b43ef46399fe115e, entries=150, sequenceid=82, filesize=11.6 K 2023-03-29 03:54:45,607 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/C/ea1581cf7b08437dbfd19c053685f48d as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/ea1581cf7b08437dbfd19c053685f48d 2023-03-29 03:54:45,612 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/ea1581cf7b08437dbfd19c053685f48d, entries=150, sequenceid=82, filesize=11.6 K 2023-03-29 03:54:45,613 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 2ed55fa255954e1310f6b9fd96e28980 in 485ms, sequenceid=82, compaction requested=false 2023-03-29 03:54:45,613 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2ed55fa255954e1310f6b9fd96e28980: 2023-03-29 03:54:45,613 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:54:45,613 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:54:45,614 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:54:45,614 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:45,614 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:54:45,614 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:54:45,616 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:45,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:45,616 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:45,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:45,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:45,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:45,617 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:45,617 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:54:45,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:45,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:45,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:45,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:45,619 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:54:45,619 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@40b88da0[Count = 0] remaining members to acquire global barrier 2023-03-29 03:54:45,619 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:54:45,619 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:45,620 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:45,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:45,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:45,620 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:54:45,620 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:54:45,620 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:54:45,621 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:45,621 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:54:45,623 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:45,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:45,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:45,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:45,623 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:54:45,623 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:45,623 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:54:45,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:45,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:45,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:45,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:45,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:45,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:45,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:45,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:54:45,627 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:54:45,627 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:54:45,627 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:54:45,627 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:54:45,627 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:54:45,627 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:54:45,629 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:45,629 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:45,629 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:45,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:45,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:45,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:45,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:45,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:45,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:45,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:45,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:45,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:45,631 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:45,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:45,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:45,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:45,632 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:45,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:45,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:45,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:45,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:45,636 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:45,636 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:45,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:45,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:45,636 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:45,636 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:45,636 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:45,636 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:54:45,636 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:54:45,636 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:54:45,636 DEBUG [Thread-1741] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:54:45,637 DEBUG [Thread-1741] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:54:45,636 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:45,636 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:45,637 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:45,637 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:45,637 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:45,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:45,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:45,737 DEBUG [Thread-1741] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:54:45,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:54:45,739 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:54:45,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:54:45,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:54:45,741 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:54:45,741 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:45,741 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:54:45,741 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:54:45,741 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:45,741 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:45,743 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:45,743 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:45,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:45,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:45,743 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:45,743 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:54:45,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:45,744 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:45,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:54:45,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:45,744 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:54:45,744 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:54:45,744 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:54:45,744 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:54:45,744 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:45,744 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:54:45,745 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:54:45,745 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:54:45,745 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:54:45,745 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. started... 2023-03-29 03:54:45,745 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2ed55fa255954e1310f6b9fd96e28980 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-29 03:54:45,745 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ed55fa255954e1310f6b9fd96e28980, store=A 2023-03-29 03:54:45,745 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:54:45,745 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ed55fa255954e1310f6b9fd96e28980, store=B 2023-03-29 03:54:45,745 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:54:45,745 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ed55fa255954e1310f6b9fd96e28980, store=C 2023-03-29 03:54:45,745 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:54:45,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on 2ed55fa255954e1310f6b9fd96e28980 2023-03-29 03:54:45,856 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. as already flushing 2023-03-29 03:54:45,868 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:45,868 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:45,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37634 deadline: 1680062145864, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:45,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37614 deadline: 1680062145865, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:45,868 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:45,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37570 deadline: 1680062145865, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:45,869 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:45,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37610 deadline: 1680062145866, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:45,870 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:45,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37572 deadline: 1680062145867, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:45,970 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:45,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37634 deadline: 1680062145969, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:45,971 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:45,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37570 deadline: 1680062145969, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:45,971 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:45,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37610 deadline: 1680062145970, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:45,972 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:45,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37614 deadline: 1680062145971, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:45,972 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:45,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37572 deadline: 1680062145972, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:46,174 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:46,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37570 deadline: 1680062146172, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:46,175 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:46,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37610 deadline: 1680062146173, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:46,175 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:46,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37614 deadline: 1680062146173, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:46,175 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:46,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37634 deadline: 1680062146173, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:46,177 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:46,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37572 deadline: 1680062146175, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:46,215 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=96 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/A/83e4364ed4be4bbabeae2258adbf0001 2023-03-29 03:54:46,430 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-03-29 03:54:46,478 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:46,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37614 deadline: 1680062146476, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:46,478 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:46,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37570 deadline: 1680062146476, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:46,479 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:46,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37610 deadline: 1680062146477, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:46,479 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:46,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37634 deadline: 1680062146478, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:46,483 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:46,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37572 deadline: 1680062146480, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:46,671 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=96 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/B/f57b2b6aed914d78b8b3510ab1b4cba6 2023-03-29 03:54:46,683 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=96 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/C/6cab4833de0746db9dab5db3d6dfc5c7 2023-03-29 03:54:46,688 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/A/83e4364ed4be4bbabeae2258adbf0001 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/83e4364ed4be4bbabeae2258adbf0001 2023-03-29 03:54:46,691 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/83e4364ed4be4bbabeae2258adbf0001, entries=150, sequenceid=96, filesize=11.6 K 2023-03-29 03:54:46,692 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/B/f57b2b6aed914d78b8b3510ab1b4cba6 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/f57b2b6aed914d78b8b3510ab1b4cba6 2023-03-29 03:54:46,695 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/f57b2b6aed914d78b8b3510ab1b4cba6, entries=150, sequenceid=96, filesize=11.6 K 2023-03-29 03:54:46,696 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/C/6cab4833de0746db9dab5db3d6dfc5c7 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/6cab4833de0746db9dab5db3d6dfc5c7 2023-03-29 03:54:46,700 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/6cab4833de0746db9dab5db3d6dfc5c7, entries=150, sequenceid=96, filesize=11.6 K 2023-03-29 03:54:46,701 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 2ed55fa255954e1310f6b9fd96e28980 in 955ms, sequenceid=96, compaction requested=true 2023-03-29 03:54:46,701 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2ed55fa255954e1310f6b9fd96e28980: 2023-03-29 03:54:46,701 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:54:46,701 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:54:46,701 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:54:46,701 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:46,701 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:54:46,701 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:54:46,703 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:46,703 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:46,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:46,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:46,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:46,703 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:46,703 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:54:46,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:46,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:46,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:46,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:46,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:46,704 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:54:46,704 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5be7e8b3[Count = 0] remaining members to acquire global barrier 2023-03-29 03:54:46,704 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:54:46,705 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:46,710 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:46,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:46,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:46,710 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:54:46,710 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:54:46,710 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:54:46,710 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:46,710 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:54:46,711 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:46,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:46,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:46,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:46,711 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:54:46,711 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:46,711 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:54:46,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:46,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:46,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:46,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:46,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:46,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:46,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:46,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:54:46,714 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:54:46,714 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:54:46,714 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:54:46,714 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:54:46,714 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:54:46,714 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:54:46,715 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:46,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:46,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:46,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:46,716 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:46,716 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:46,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:46,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:46,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:46,716 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:46,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:46,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:46,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:46,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:46,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:46,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:46,717 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:46,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:46,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:46,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:46,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:46,720 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:46,720 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:46,720 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:46,720 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:46,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:46,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:46,720 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:46,721 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:46,721 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:54:46,721 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:54:46,721 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:46,721 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:46,721 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:46,721 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:46,721 DEBUG [Thread-1741] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:54:46,721 DEBUG [Thread-1741] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:54:46,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:46,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:46,821 DEBUG [Thread-1741] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:54:46,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:54:46,823 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:54:46,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:54:46,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:54:46,826 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:54:46,826 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:46,826 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:54:46,826 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:54:46,827 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:46,827 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:46,828 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:46,828 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:46,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:46,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:46,829 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:46,829 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:54:46,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:46,829 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:46,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:54:46,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:46,830 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:54:46,830 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:54:46,830 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:54:46,830 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:54:46,830 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:46,830 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:54:46,830 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:54:46,830 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:54:46,830 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:54:46,830 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. started... 2023-03-29 03:54:46,830 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2ed55fa255954e1310f6b9fd96e28980 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-29 03:54:46,831 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ed55fa255954e1310f6b9fd96e28980, store=A 2023-03-29 03:54:46,831 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:54:46,831 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ed55fa255954e1310f6b9fd96e28980, store=B 2023-03-29 03:54:46,831 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:54:46,831 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ed55fa255954e1310f6b9fd96e28980, store=C 2023-03-29 03:54:46,831 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:54:46,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on 2ed55fa255954e1310f6b9fd96e28980 2023-03-29 03:54:46,982 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. as already flushing 2023-03-29 03:54:46,993 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:46,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37634 deadline: 1680062146989, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:46,993 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:46,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37570 deadline: 1680062146990, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:46,994 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:46,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37614 deadline: 1680062146991, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:46,995 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:46,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37610 deadline: 1680062146994, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:46,995 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:46,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37572 deadline: 1680062146994, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:47,095 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:47,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37570 deadline: 1680062147094, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:47,096 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:47,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37634 deadline: 1680062147094, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:47,097 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:47,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37614 deadline: 1680062147095, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:47,097 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:47,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37610 deadline: 1680062147096, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:47,097 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:47,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37572 deadline: 1680062147096, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:47,238 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=115 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/A/a6b5f9126e2c4320ac1f6421a8063c4b 2023-03-29 03:54:47,299 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:47,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37570 deadline: 1680062147297, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:47,299 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:47,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37634 deadline: 1680062147298, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:47,299 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:47,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37614 deadline: 1680062147298, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:47,299 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:47,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37610 deadline: 1680062147298, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:47,299 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:47,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37572 deadline: 1680062147298, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:47,601 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:47,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37610 deadline: 1680062147600, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:47,602 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:47,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37570 deadline: 1680062147601, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:47,602 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:47,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37634 deadline: 1680062147601, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:47,602 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:47,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37614 deadline: 1680062147601, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:47,604 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:47,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37572 deadline: 1680062147602, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:47,648 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=115 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/B/02c751b5b36240969d39909a28ec3412 2023-03-29 03:54:48,058 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=115 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/C/a0723fc844e6448dbec6821a9d0a2493 2023-03-29 03:54:48,065 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/A/a6b5f9126e2c4320ac1f6421a8063c4b as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/a6b5f9126e2c4320ac1f6421a8063c4b 2023-03-29 03:54:48,071 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/a6b5f9126e2c4320ac1f6421a8063c4b, entries=150, sequenceid=115, filesize=11.6 K 2023-03-29 03:54:48,072 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/B/02c751b5b36240969d39909a28ec3412 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/02c751b5b36240969d39909a28ec3412 2023-03-29 03:54:48,076 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/02c751b5b36240969d39909a28ec3412, entries=150, sequenceid=115, filesize=11.6 K 2023-03-29 03:54:48,077 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/C/a0723fc844e6448dbec6821a9d0a2493 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/a0723fc844e6448dbec6821a9d0a2493 2023-03-29 03:54:48,080 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/a0723fc844e6448dbec6821a9d0a2493, entries=150, sequenceid=115, filesize=11.6 K 2023-03-29 03:54:48,081 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 2ed55fa255954e1310f6b9fd96e28980 in 1251ms, sequenceid=115, compaction requested=true 2023-03-29 03:54:48,081 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2ed55fa255954e1310f6b9fd96e28980: 2023-03-29 03:54:48,081 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:54:48,081 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:54:48,081 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:54:48,081 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:48,081 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:54:48,081 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:54:48,083 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:48,083 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:48,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:48,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:48,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:48,083 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:48,083 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:54:48,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:48,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:48,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:48,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:48,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:48,085 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:54:48,085 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@705db8d9[Count = 0] remaining members to acquire global barrier 2023-03-29 03:54:48,085 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:54:48,085 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:48,086 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:48,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:48,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:48,086 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:54:48,086 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:48,086 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:54:48,086 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:54:48,086 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:54:48,088 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:48,088 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:54:48,089 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:48,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:48,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:48,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:48,089 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:54:48,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:48,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:48,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:48,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:48,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:48,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:48,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:48,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:54:48,091 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:54:48,091 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:54:48,091 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:54:48,091 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:54:48,091 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:54:48,091 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:54:48,093 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:48,093 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:48,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:48,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:48,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:48,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:48,093 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:48,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:48,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:48,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:48,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:48,093 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:48,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:48,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:48,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:48,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:48,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:48,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:48,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:48,095 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:48,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:48,097 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:48,097 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:48,097 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:48,097 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:48,097 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:48,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:48,097 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:54:48,098 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:54:48,097 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:48,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:48,097 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:48,098 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:48,098 DEBUG [Thread-1741] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:54:48,098 DEBUG [Thread-1741] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:54:48,098 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:48,098 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:48,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:48,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:48,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on 2ed55fa255954e1310f6b9fd96e28980 2023-03-29 03:54:48,105 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2ed55fa255954e1310f6b9fd96e28980 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-03-29 03:54:48,106 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ed55fa255954e1310f6b9fd96e28980, store=A 2023-03-29 03:54:48,106 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:54:48,106 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ed55fa255954e1310f6b9fd96e28980, store=B 2023-03-29 03:54:48,106 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:54:48,106 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ed55fa255954e1310f6b9fd96e28980, store=C 2023-03-29 03:54:48,106 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:54:48,113 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=131 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/A/92624db90abe44d394a95b0e9b0577be 2023-03-29 03:54:48,119 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:48,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37614 deadline: 1680062148116, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:48,119 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:48,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37572 deadline: 1680062148116, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:48,120 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:48,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37570 deadline: 1680062148117, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:48,122 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:48,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37610 deadline: 1680062148119, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:48,122 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:48,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37634 deadline: 1680062148119, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:48,198 DEBUG [Thread-1741] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:54:48,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:54:48,200 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:54:48,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:54:48,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:54:48,201 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:54:48,201 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:48,201 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:54:48,201 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:54:48,202 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:48,202 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:48,203 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:48,203 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:48,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:48,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:48,203 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:48,203 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:54:48,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:48,204 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:48,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:54:48,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:48,204 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:54:48,204 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:54:48,204 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:54:48,204 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:54:48,204 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:48,204 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:54:48,204 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:54:48,204 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:54:48,205 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:54:48,205 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. started... 2023-03-29 03:54:48,205 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. as already flushing 2023-03-29 03:54:48,205 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:54:48,205 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:54:48,205 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:54:48,205 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:48,205 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:54:48,205 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:54:48,207 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:48,207 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:48,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:48,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:48,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:48,208 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:48,208 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:54:48,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:48,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:48,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:48,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:48,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:48,209 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:54:48,209 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3220013[Count = 0] remaining members to acquire global barrier 2023-03-29 03:54:48,209 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:54:48,209 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:48,210 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:48,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:48,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:48,210 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:54:48,210 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:48,210 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:54:48,210 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:54:48,210 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:54:48,212 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:48,212 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:54:48,212 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:48,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:48,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:48,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:48,212 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:54:48,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:48,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:48,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:48,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:48,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:48,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:48,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:48,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:54:48,214 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:54:48,214 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:54:48,214 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:54:48,214 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:54:48,214 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:54:48,214 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:54:48,216 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:48,216 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:48,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:48,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:48,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:48,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:48,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:48,216 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:48,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:48,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:48,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:48,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:48,217 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:48,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:48,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:48,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:48,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:48,218 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:48,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:48,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:48,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:48,221 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:48,221 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:48,221 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:48,221 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:48,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:48,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:48,222 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:54:48,222 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:54:48,221 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:48,222 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:48,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:48,222 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:48,222 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:48,222 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:48,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:48,222 DEBUG [Thread-1741] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:54:48,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37572 deadline: 1680062148220, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:48,222 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:48,222 DEBUG [Thread-1741] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:54:48,222 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:48,223 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:48,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37614 deadline: 1680062148220, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:48,223 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:48,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37570 deadline: 1680062148220, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:48,224 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:48,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37610 deadline: 1680062148223, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:48,225 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:48,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37634 deadline: 1680062148223, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:48,322 DEBUG [Thread-1741] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:54:48,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:54:48,324 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:54:48,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:54:48,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:54:48,326 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:54:48,326 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:48,326 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:54:48,326 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:54:48,326 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:48,326 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:48,328 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:48,328 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:48,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:48,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:48,328 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:48,328 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:54:48,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:48,328 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:48,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:54:48,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:48,329 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:54:48,329 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:54:48,329 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:54:48,329 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:54:48,329 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:48,329 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:54:48,329 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:54:48,330 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:54:48,330 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:54:48,330 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. started... 2023-03-29 03:54:48,330 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. as already flushing 2023-03-29 03:54:48,330 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:54:48,330 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:54:48,330 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:54:48,330 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:48,330 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:54:48,330 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:54:48,331 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:48,331 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:48,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:48,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:48,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:48,332 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:48,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:48,332 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:54:48,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:48,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:48,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:48,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:48,333 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:54:48,333 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@773485a[Count = 0] remaining members to acquire global barrier 2023-03-29 03:54:48,333 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:54:48,333 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:48,334 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:48,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:48,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:48,334 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:54:48,335 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:54:48,335 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:54:48,335 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:48,335 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:54:48,340 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:54:48,340 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:48,340 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:48,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:48,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:48,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:48,340 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:54:48,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:48,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:48,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:48,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:48,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:48,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:48,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:48,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:54:48,346 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:54:48,346 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:54:48,346 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:54:48,346 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:54:48,346 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:54:48,346 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:54:48,348 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:48,348 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:48,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:48,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:48,348 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:48,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:48,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:48,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:48,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:48,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:48,349 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:48,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:48,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:48,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:48,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:48,351 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:48,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:48,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:48,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:48,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:48,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:48,353 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:48,353 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:48,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:48,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:48,353 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:48,353 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:48,353 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:48,353 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:48,353 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:48,353 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:48,354 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:48,354 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:48,354 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:54:48,354 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:54:48,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:48,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:48,354 DEBUG [Thread-1741] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:54:48,354 DEBUG [Thread-1741] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:54:48,354 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:54:48,425 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:48,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37572 deadline: 1680062148423, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:48,426 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:48,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37614 deadline: 1680062148423, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:48,427 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:48,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37570 deadline: 1680062148425, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:48,429 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:48,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37610 deadline: 1680062148425, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:48,429 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:48,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37634 deadline: 1680062148426, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:48,454 DEBUG [Thread-1741] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:54:48,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:54:48,456 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:54:48,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:54:48,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:54:48,458 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:54:48,458 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:48,458 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:54:48,458 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:54:48,458 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:48,458 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:48,460 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:48,460 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:48,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:48,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:48,461 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:48,461 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:54:48,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:48,461 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:48,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:54:48,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:48,461 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:54:48,462 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:54:48,462 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:54:48,462 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:54:48,462 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:48,462 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:54:48,462 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:54:48,462 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:54:48,462 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:54:48,462 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. started... 2023-03-29 03:54:48,462 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. as already flushing 2023-03-29 03:54:48,462 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:54:48,462 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:54:48,462 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:54:48,462 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:48,462 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:54:48,463 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:54:48,464 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:48,464 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:48,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:48,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:48,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:48,464 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:48,464 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:54:48,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:48,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:48,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:48,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:48,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:48,466 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:54:48,466 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@63359a77[Count = 0] remaining members to acquire global barrier 2023-03-29 03:54:48,466 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:54:48,466 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:48,467 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:48,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:48,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:48,467 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:54:48,467 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:54:48,467 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:54:48,467 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:48,467 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:54:48,468 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:48,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:48,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:48,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:48,468 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:54:48,469 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:48,469 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:54:48,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:48,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:48,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:48,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:48,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:48,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:48,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:48,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:54:48,471 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:54:48,471 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:54:48,471 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:54:48,471 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:54:48,471 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:54:48,471 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:54:48,473 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:48,473 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:48,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:48,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:48,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:48,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:48,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:48,473 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:48,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:48,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:48,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:48,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:48,474 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:48,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:48,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:48,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:48,475 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:48,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:48,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:48,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:48,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:48,482 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:48,482 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:48,482 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:48,482 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:54:48,482 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:54:48,483 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:48,483 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:48,483 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:48,483 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:48,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:48,483 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:48,483 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:54:48,483 DEBUG [Thread-1741] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:54:48,483 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:48,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:48,483 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:48,483 DEBUG [Thread-1741] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:54:48,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:48,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:48,523 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=131 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/B/f1f8d58b22c04f5ea07843af4b74bedf 2023-03-29 03:54:48,583 DEBUG [Thread-1741] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:54:48,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:54:48,586 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:54:48,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:54:48,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:54:48,587 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:54:48,587 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:48,587 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:54:48,587 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:54:48,588 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:48,588 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:48,589 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:48,589 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:48,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:48,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:48,589 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:48,589 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:54:48,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:48,590 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:48,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:54:48,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:48,590 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:54:48,590 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:54:48,590 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:54:48,590 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:54:48,590 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:48,590 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:54:48,591 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:54:48,591 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:54:48,591 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:54:48,591 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. started... 2023-03-29 03:54:48,591 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. as already flushing 2023-03-29 03:54:48,591 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:54:48,591 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:54:48,591 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:54:48,591 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:48,591 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:54:48,591 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:54:48,593 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:48,593 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:48,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:48,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:48,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:48,593 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:48,593 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:54:48,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:48,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:48,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:48,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:48,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:48,594 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:54:48,594 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7eb36db8[Count = 0] remaining members to acquire global barrier 2023-03-29 03:54:48,594 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:54:48,594 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:48,597 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:48,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:48,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:48,597 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:54:48,597 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:48,597 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:54:48,597 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:54:48,597 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:54:48,598 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:54:48,598 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:48,598 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:48,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:48,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:48,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:48,599 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:54:48,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:48,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:48,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:48,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:48,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:48,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:48,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:48,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:54:48,601 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:54:48,601 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:54:48,601 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:54:48,601 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:54:48,601 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:54:48,601 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:54:48,602 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:48,602 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:48,603 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:48,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:48,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:48,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:48,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:48,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:48,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:48,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:48,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:48,603 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:48,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:48,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:48,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:48,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:48,604 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:48,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:48,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:48,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:48,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:48,606 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:48,606 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:48,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:48,606 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:48,606 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:48,607 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:54:48,607 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:54:48,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:48,607 DEBUG [Thread-1741] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:54:48,607 DEBUG [Thread-1741] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:54:48,607 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:48,607 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:48,607 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:48,607 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:48,607 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:48,607 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:48,607 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:54:48,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:48,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:48,707 DEBUG [Thread-1741] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:54:48,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:54:48,709 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:54:48,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:54:48,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:54:48,711 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:54:48,711 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:48,711 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:54:48,711 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:54:48,711 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:48,711 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:48,713 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:48,713 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:48,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:48,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:48,714 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:48,714 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:54:48,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:48,714 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:48,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:54:48,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:48,715 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:54:48,715 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:54:48,715 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:54:48,715 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:54:48,715 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:48,715 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:54:48,715 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:54:48,715 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:54:48,715 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:54:48,715 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. started... 2023-03-29 03:54:48,715 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. as already flushing 2023-03-29 03:54:48,716 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:54:48,716 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:54:48,716 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:54:48,716 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:48,716 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:54:48,716 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:54:48,717 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:48,717 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:48,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:48,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:48,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:48,717 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:48,718 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:54:48,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:48,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:48,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:48,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:48,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:48,719 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:54:48,719 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6d02c8e5[Count = 0] remaining members to acquire global barrier 2023-03-29 03:54:48,719 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:54:48,719 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:48,720 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:48,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:48,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:48,720 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:54:48,720 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:54:48,720 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:54:48,720 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:48,721 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:54:48,722 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:48,722 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:54:48,722 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:48,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:48,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:48,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:48,722 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:54:48,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:48,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:48,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:48,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:48,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:48,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:48,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:48,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:54:48,724 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:54:48,724 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:54:48,724 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:54:48,724 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:54:48,724 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:54:48,724 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:54:48,726 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:48,726 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:48,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:48,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:48,726 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:48,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:48,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:48,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:48,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:48,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:48,727 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:48,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:48,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:48,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:48,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:48,728 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:48,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:48,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:48,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:48,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:48,730 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:48,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37614 deadline: 1680062148728, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:48,730 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:48,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37572 deadline: 1680062148728, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:48,730 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:48,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37570 deadline: 1680062148729, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:48,732 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:48,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37610 deadline: 1680062148731, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:48,733 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:48,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37634 deadline: 1680062148731, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:48,734 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:48,734 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:48,734 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:48,734 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:48,734 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:48,734 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:48,734 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:48,734 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:48,734 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:48,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:48,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:48,734 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:54:48,734 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:54:48,734 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:54:48,734 DEBUG [Thread-1741] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:54:48,734 DEBUG [Thread-1741] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:54:48,734 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:48,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:48,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:48,834 DEBUG [Thread-1741] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:54:48,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:54:48,837 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:54:48,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:54:48,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:54:48,838 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:54:48,838 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:48,838 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:54:48,838 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:54:48,838 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:48,838 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:48,840 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:48,840 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:48,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:48,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:48,840 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:48,840 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:54:48,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:48,841 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:48,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:54:48,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:48,841 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:54:48,841 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:54:48,841 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:54:48,841 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:54:48,841 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:48,841 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:54:48,842 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:54:48,842 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:54:48,842 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. started... 2023-03-29 03:54:48,842 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:54:48,842 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. as already flushing 2023-03-29 03:54:48,842 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:54:48,842 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:54:48,842 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:54:48,842 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:48,842 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:54:48,842 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:54:48,844 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:48,844 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:48,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:48,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:48,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:48,844 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:48,844 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:54:48,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:48,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:48,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:48,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:48,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:48,846 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:54:48,846 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70ae6c87[Count = 0] remaining members to acquire global barrier 2023-03-29 03:54:48,846 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:54:48,846 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:48,848 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:48,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:48,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:48,848 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:54:48,848 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:54:48,848 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:54:48,848 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:48,848 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:54:48,849 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:54:48,850 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:48,849 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:48,850 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:54:48,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:48,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:48,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:48,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:48,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:48,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:48,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:48,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:48,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:48,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:48,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:54:48,852 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:54:48,852 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:54:48,852 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:54:48,852 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:54:48,852 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:54:48,852 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:54:48,854 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:48,854 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:48,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:48,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:48,854 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:48,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:48,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:48,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:48,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:48,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:48,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:48,855 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:48,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:48,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:48,856 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:48,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:48,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:48,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:48,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:48,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:48,859 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:48,859 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:48,859 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:48,859 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:48,859 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:54:48,859 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:54:48,859 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:48,859 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:48,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:48,859 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:48,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:48,859 DEBUG [Thread-1741] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:54:48,859 DEBUG [Thread-1741] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:54:48,859 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:54:48,859 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:48,860 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:48,859 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:48,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:48,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:48,934 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=131 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/C/5b7c7b9c1c814a2283cb69858846a885 2023-03-29 03:54:48,939 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/A/92624db90abe44d394a95b0e9b0577be as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/92624db90abe44d394a95b0e9b0577be 2023-03-29 03:54:48,943 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/92624db90abe44d394a95b0e9b0577be, entries=150, sequenceid=131, filesize=11.7 K 2023-03-29 03:54:48,944 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/B/f1f8d58b22c04f5ea07843af4b74bedf as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/f1f8d58b22c04f5ea07843af4b74bedf 2023-03-29 03:54:48,948 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/f1f8d58b22c04f5ea07843af4b74bedf, entries=150, sequenceid=131, filesize=11.7 K 2023-03-29 03:54:48,949 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/C/5b7c7b9c1c814a2283cb69858846a885 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/5b7c7b9c1c814a2283cb69858846a885 2023-03-29 03:54:48,953 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/5b7c7b9c1c814a2283cb69858846a885, entries=150, sequenceid=131, filesize=11.7 K 2023-03-29 03:54:48,954 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 2ed55fa255954e1310f6b9fd96e28980 in 849ms, sequenceid=131, compaction requested=true 2023-03-29 03:54:48,954 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2ed55fa255954e1310f6b9fd96e28980: 2023-03-29 03:54:48,954 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-29 03:54:48,954 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:54:48,954 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:54:48,954 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:54:48,954 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-29 03:54:48,960 DEBUG [Thread-1741] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:54:48,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:54:48,962 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:54:48,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:54:48,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:54:48,964 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:54:48,964 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:48,964 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:54:48,964 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:54:48,964 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:48,964 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:48,965 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:48,966 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:48,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:48,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:48,966 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:48,966 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:54:48,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:48,966 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:48,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:54:48,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:48,967 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:54:48,967 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:54:48,967 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:54:48,967 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:54:48,967 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:48,967 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:54:48,968 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:54:48,968 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:54:48,968 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:54:48,968 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. started... 2023-03-29 03:54:48,970 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2ed55fa255954e1310f6b9fd96e28980 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-03-29 03:54:48,970 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59583 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-29 03:54:48,970 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ed55fa255954e1310f6b9fd96e28980, store=A 2023-03-29 03:54:48,970 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:54:48,970 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ed55fa255954e1310f6b9fd96e28980, store=B 2023-03-29 03:54:48,970 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59583 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-29 03:54:48,970 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1912): 2ed55fa255954e1310f6b9fd96e28980/B is initiating minor compaction (all files) 2023-03-29 03:54:48,970 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:54:48,971 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2ed55fa255954e1310f6b9fd96e28980/B in TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:54:48,971 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1912): 2ed55fa255954e1310f6b9fd96e28980/A is initiating minor compaction (all files) 2023-03-29 03:54:48,971 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ed55fa255954e1310f6b9fd96e28980, store=C 2023-03-29 03:54:48,971 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2ed55fa255954e1310f6b9fd96e28980/A in TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:54:48,971 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/2e93d88526c5484fbd124c2e5ffd8ed0, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/37cbfd98fc4549a0b43ef46399fe115e, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/f57b2b6aed914d78b8b3510ab1b4cba6, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/02c751b5b36240969d39909a28ec3412, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/f1f8d58b22c04f5ea07843af4b74bedf] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp, totalSize=58.2 K 2023-03-29 03:54:48,971 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/fb37541f10a54c0aa4f04af8de268e0c, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/b884545047f64c17b7949a2c6b266711, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/83e4364ed4be4bbabeae2258adbf0001, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/a6b5f9126e2c4320ac1f6421a8063c4b, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/92624db90abe44d394a95b0e9b0577be] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp, totalSize=58.2 K 2023-03-29 03:54:48,971 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 2e93d88526c5484fbd124c2e5ffd8ed0, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=61, earliestPutTs=1680062084540 2023-03-29 03:54:48,972 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting fb37541f10a54c0aa4f04af8de268e0c, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=61, earliestPutTs=1680062084540 2023-03-29 03:54:48,971 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:54:48,972 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 37cbfd98fc4549a0b43ef46399fe115e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1680062084867 2023-03-29 03:54:48,972 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting b884545047f64c17b7949a2c6b266711, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1680062084867 2023-03-29 03:54:48,973 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting f57b2b6aed914d78b8b3510ab1b4cba6, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=96, earliestPutTs=1680062085188 2023-03-29 03:54:48,973 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 83e4364ed4be4bbabeae2258adbf0001, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=96, earliestPutTs=1680062085188 2023-03-29 03:54:48,973 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 02c751b5b36240969d39909a28ec3412, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=115, earliestPutTs=1680062085863 2023-03-29 03:54:48,974 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting a6b5f9126e2c4320ac1f6421a8063c4b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=115, earliestPutTs=1680062085863 2023-03-29 03:54:48,974 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting f1f8d58b22c04f5ea07843af4b74bedf, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=131, earliestPutTs=1680062086987 2023-03-29 03:54:48,974 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 92624db90abe44d394a95b0e9b0577be, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=131, earliestPutTs=1680062086987 2023-03-29 03:54:48,994 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2ed55fa255954e1310f6b9fd96e28980#A#compaction#245 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:54:48,996 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] throttle.PressureAwareThroughputController(145): 2ed55fa255954e1310f6b9fd96e28980#B#compaction#246 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:54:49,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on 2ed55fa255954e1310f6b9fd96e28980 2023-03-29 03:54:49,236 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. as already flushing 2023-03-29 03:54:49,246 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:49,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37610 deadline: 1680062149243, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:49,246 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:49,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37570 deadline: 1680062149243, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:49,247 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:49,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37572 deadline: 1680062149244, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:49,248 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:49,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37634 deadline: 1680062149245, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:49,248 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:49,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37614 deadline: 1680062149246, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:49,349 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:49,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37610 deadline: 1680062149347, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:49,349 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:49,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37570 deadline: 1680062149347, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:49,350 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:49,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37572 deadline: 1680062149348, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:49,350 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:49,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37634 deadline: 1680062149349, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:49,351 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:49,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37614 deadline: 1680062149349, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:49,404 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=147 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/A/560f1517f0c4405c8668da436fbfeb5a 2023-03-29 03:54:49,423 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/B/b7e543181c5f490e83467ca868a07b8e as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/b7e543181c5f490e83467ca868a07b8e 2023-03-29 03:54:49,426 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/A/39b6b51e018d43d19d6c3de0f0f27f0d as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/39b6b51e018d43d19d6c3de0f0f27f0d 2023-03-29 03:54:49,429 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 2ed55fa255954e1310f6b9fd96e28980/B of 2ed55fa255954e1310f6b9fd96e28980 into b7e543181c5f490e83467ca868a07b8e(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:54:49,429 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2ed55fa255954e1310f6b9fd96e28980: 2023-03-29 03:54:49,429 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980., storeName=2ed55fa255954e1310f6b9fd96e28980/B, priority=11, startTime=1680062088954; duration=0sec 2023-03-29 03:54:49,429 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:54:49,429 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-29 03:54:49,431 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59583 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-29 03:54:49,431 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1912): 2ed55fa255954e1310f6b9fd96e28980/C is initiating minor compaction (all files) 2023-03-29 03:54:49,431 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2ed55fa255954e1310f6b9fd96e28980/C in TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:54:49,431 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/91ee23eb360842a29ac70105fa934f9f, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/ea1581cf7b08437dbfd19c053685f48d, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/6cab4833de0746db9dab5db3d6dfc5c7, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/a0723fc844e6448dbec6821a9d0a2493, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/5b7c7b9c1c814a2283cb69858846a885] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp, totalSize=58.2 K 2023-03-29 03:54:49,432 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 91ee23eb360842a29ac70105fa934f9f, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=61, earliestPutTs=1680062084540 2023-03-29 03:54:49,432 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 2ed55fa255954e1310f6b9fd96e28980/A of 2ed55fa255954e1310f6b9fd96e28980 into 39b6b51e018d43d19d6c3de0f0f27f0d(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:54:49,432 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2ed55fa255954e1310f6b9fd96e28980: 2023-03-29 03:54:49,432 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980., storeName=2ed55fa255954e1310f6b9fd96e28980/A, priority=11, startTime=1680062088954; duration=0sec 2023-03-29 03:54:49,432 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:54:49,432 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting ea1581cf7b08437dbfd19c053685f48d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1680062084867 2023-03-29 03:54:49,432 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 6cab4833de0746db9dab5db3d6dfc5c7, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=96, earliestPutTs=1680062085188 2023-03-29 03:54:49,433 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting a0723fc844e6448dbec6821a9d0a2493, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=115, earliestPutTs=1680062085863 2023-03-29 03:54:49,433 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 5b7c7b9c1c814a2283cb69858846a885, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=131, earliestPutTs=1680062086987 2023-03-29 03:54:49,440 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] throttle.PressureAwareThroughputController(145): 2ed55fa255954e1310f6b9fd96e28980#C#compaction#248 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:54:49,448 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/C/77de40d9d4b2449ab7e381b4182db937 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/77de40d9d4b2449ab7e381b4182db937 2023-03-29 03:54:49,455 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 2ed55fa255954e1310f6b9fd96e28980/C of 2ed55fa255954e1310f6b9fd96e28980 into 77de40d9d4b2449ab7e381b4182db937(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:54:49,455 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2ed55fa255954e1310f6b9fd96e28980: 2023-03-29 03:54:49,455 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980., storeName=2ed55fa255954e1310f6b9fd96e28980/C, priority=11, startTime=1680062088954; duration=0sec 2023-03-29 03:54:49,455 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:54:49,552 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:49,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37570 deadline: 1680062149550, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:49,552 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:49,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37572 deadline: 1680062149550, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:49,552 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:49,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37610 deadline: 1680062149551, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:49,553 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:49,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37634 deadline: 1680062149551, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:49,553 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:49,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37614 deadline: 1680062149551, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:49,813 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=147 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/B/71ca5d31c1244abea369822ff32bd33b 2023-03-29 03:54:49,855 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:49,855 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:49,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37572 deadline: 1680062149853, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:49,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37610 deadline: 1680062149853, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:49,855 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:49,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37614 deadline: 1680062149854, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:49,855 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:49,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37570 deadline: 1680062149855, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:49,857 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:49,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37634 deadline: 1680062149856, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:50,223 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=147 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/C/2ac363cfe65a4ad3b584c6745815d5e8 2023-03-29 03:54:50,227 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/A/560f1517f0c4405c8668da436fbfeb5a as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/560f1517f0c4405c8668da436fbfeb5a 2023-03-29 03:54:50,231 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/560f1517f0c4405c8668da436fbfeb5a, entries=150, sequenceid=147, filesize=11.8 K 2023-03-29 03:54:50,232 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/B/71ca5d31c1244abea369822ff32bd33b as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/71ca5d31c1244abea369822ff32bd33b 2023-03-29 03:54:50,235 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/71ca5d31c1244abea369822ff32bd33b, entries=150, sequenceid=147, filesize=11.8 K 2023-03-29 03:54:50,236 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/C/2ac363cfe65a4ad3b584c6745815d5e8 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/2ac363cfe65a4ad3b584c6745815d5e8 2023-03-29 03:54:50,239 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/2ac363cfe65a4ad3b584c6745815d5e8, entries=150, sequenceid=147, filesize=11.8 K 2023-03-29 03:54:50,240 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 2ed55fa255954e1310f6b9fd96e28980 in 1272ms, sequenceid=147, compaction requested=false 2023-03-29 03:54:50,240 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2ed55fa255954e1310f6b9fd96e28980: 2023-03-29 03:54:50,240 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:54:50,240 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:54:50,240 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:54:50,240 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:50,240 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:54:50,240 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:54:50,242 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:50,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:50,242 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:50,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:50,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:50,242 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:50,242 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:54:50,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:50,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:50,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:50,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:50,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:50,244 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:54:50,244 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3f07b6c7[Count = 0] remaining members to acquire global barrier 2023-03-29 03:54:50,244 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:54:50,244 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:50,245 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:50,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:50,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:50,245 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:54:50,245 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:54:50,245 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:54:50,245 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:50,245 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:54:50,247 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:54:50,247 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:50,247 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:50,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:50,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:50,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:50,247 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:54:50,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:50,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:50,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:50,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:50,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:50,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:50,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:50,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:54:50,250 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:54:50,250 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:54:50,250 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:54:50,250 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:54:50,250 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:54:50,250 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:54:50,252 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:50,252 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:50,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:50,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:50,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:50,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:50,252 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:50,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:50,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:50,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:50,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:50,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:50,253 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:50,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:50,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:50,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:50,254 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:50,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:50,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:50,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:50,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:50,257 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:50,257 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:50,257 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:50,257 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:50,257 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:50,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:50,257 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:54:50,257 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:54:50,257 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:50,257 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:50,257 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:50,257 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:50,257 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:50,257 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:54:50,257 DEBUG [Thread-1741] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:54:50,257 DEBUG [Thread-1741] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:54:50,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:50,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:50,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:50,357 DEBUG [Thread-1741] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:54:50,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on 2ed55fa255954e1310f6b9fd96e28980 2023-03-29 03:54:50,358 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2ed55fa255954e1310f6b9fd96e28980 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-03-29 03:54:50,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:54:50,358 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ed55fa255954e1310f6b9fd96e28980, store=A 2023-03-29 03:54:50,358 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:54:50,358 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ed55fa255954e1310f6b9fd96e28980, store=B 2023-03-29 03:54:50,358 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:54:50,358 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ed55fa255954e1310f6b9fd96e28980, store=C 2023-03-29 03:54:50,358 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:54:50,360 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:54:50,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:54:50,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:54:50,361 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:54:50,361 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:50,361 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:54:50,361 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:54:50,362 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:50,362 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:50,364 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:50,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:50,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:50,364 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:50,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:50,365 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:50,365 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:54:50,365 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:50,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:54:50,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:50,366 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:54:50,366 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:54:50,366 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:54:50,366 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:54:50,367 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:50,367 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:54:50,367 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:54:50,367 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:54:50,367 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:54:50,367 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. started... 2023-03-29 03:54:50,367 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. as already flushing 2023-03-29 03:54:50,367 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:54:50,367 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:54:50,367 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:54:50,367 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:50,367 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:54:50,367 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:54:50,369 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:50,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37610 deadline: 1680062150366, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:50,369 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:50,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37614 deadline: 1680062150367, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:50,369 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:50,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37572 deadline: 1680062150367, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:50,370 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:50,370 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:50,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:50,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:50,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:50,370 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:50,370 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:54:50,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:50,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:50,371 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:50,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37634 deadline: 1680062150368, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:50,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:50,371 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:50,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37570 deadline: 1680062150369, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:50,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:50,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:50,372 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:54:50,372 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5abcd232[Count = 0] remaining members to acquire global barrier 2023-03-29 03:54:50,372 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:54:50,372 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:50,373 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:50,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:50,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:50,373 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:54:50,373 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:54:50,373 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:50,374 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:54:50,374 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:54:50,375 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:50,375 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:54:50,375 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:50,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:50,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:50,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:50,375 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:54:50,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:50,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:50,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:50,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:50,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:50,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:50,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:50,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:54:50,377 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:54:50,377 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:54:50,377 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:54:50,377 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:54:50,377 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:54:50,377 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:54:50,379 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:50,379 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:50,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:50,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:50,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:50,379 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:50,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:50,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:50,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:50,379 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:50,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:50,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:50,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:50,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:50,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:50,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:50,380 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:50,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:50,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:50,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:50,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:50,386 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:50,386 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:50,386 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:50,386 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:54:50,386 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:50,386 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:54:50,386 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:54:50,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:50,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:50,386 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:50,386 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:50,387 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:50,387 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:50,387 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:50,387 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:50,387 DEBUG [Thread-1741] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:54:50,387 DEBUG [Thread-1741] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:54:50,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:50,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:50,471 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:50,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37610 deadline: 1680062150470, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:50,471 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:50,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37614 deadline: 1680062150470, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:50,471 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:50,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37572 deadline: 1680062150470, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:50,473 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:50,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37634 deadline: 1680062150472, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:50,474 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:50,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37570 deadline: 1680062150472, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:50,487 DEBUG [Thread-1741] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:54:50,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:54:50,489 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:54:50,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:54:50,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:54:50,491 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:54:50,491 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:50,491 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:54:50,491 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:54:50,492 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:50,492 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:50,493 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:50,493 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:50,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:50,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:50,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:50,494 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:50,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:54:50,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:50,494 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:54:50,495 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:54:50,495 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:54:50,495 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:54:50,495 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:50,495 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:54:50,495 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:50,500 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:54:50,500 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:54:50,500 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:54:50,500 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:54:50,500 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. started... 2023-03-29 03:54:50,500 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. as already flushing 2023-03-29 03:54:50,500 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:54:50,500 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:54:50,500 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:54:50,500 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:50,500 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:54:50,500 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:54:50,502 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:50,502 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:50,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:50,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:50,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:50,503 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:50,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:50,503 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:54:50,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:50,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:50,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:50,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:50,505 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:54:50,505 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@21826c60[Count = 0] remaining members to acquire global barrier 2023-03-29 03:54:50,505 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:54:50,505 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:50,507 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:50,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:50,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:50,508 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:54:50,508 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:54:50,508 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:54:50,509 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:50,509 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:54:50,509 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:50,509 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:54:50,510 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:54:50,510 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:54:50,510 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:54:50,510 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:54:50,510 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:54:50,510 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:54:50,510 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:54:50,511 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:50,511 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:50,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:50,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:50,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:50,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:50,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:50,511 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:50,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:50,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:50,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:50,512 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:50,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:50,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:50,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:50,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:50,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:50,513 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:50,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:50,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:50,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:50,516 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:50,516 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:50,516 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:50,516 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:50,516 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:50,516 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:50,516 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:54:50,516 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:54:50,516 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:54:50,516 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:50,516 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:50,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:50,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:50,517 DEBUG [Thread-1741] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:54:50,517 DEBUG [Thread-1741] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:54:50,516 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:50,517 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:50,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:50,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:50,617 DEBUG [Thread-1741] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:54:50,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:54:50,619 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:54:50,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:54:50,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:54:50,621 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:54:50,621 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:50,622 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:54:50,622 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:54:50,622 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:50,622 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:50,627 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:50,627 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:50,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:50,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:50,628 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:50,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:50,628 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:54:50,628 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:50,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:54:50,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:50,629 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:54:50,630 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:54:50,630 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:54:50,630 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:54:50,630 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:50,630 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:54:50,636 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:54:50,636 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:54:50,636 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:54:50,636 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. started... 2023-03-29 03:54:50,636 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. as already flushing 2023-03-29 03:54:50,636 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:54:50,636 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:54:50,636 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:54:50,636 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:50,636 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:54:50,636 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:54:50,638 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:50,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:50,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:50,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:50,638 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:50,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:50,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:50,639 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:50,639 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:54:50,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:50,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:50,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:50,640 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:54:50,640 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@22cbe683[Count = 0] remaining members to acquire global barrier 2023-03-29 03:54:50,640 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:54:50,640 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:50,642 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:50,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:50,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:50,642 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:54:50,642 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:54:50,642 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:54:50,643 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:50,643 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:54:50,644 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:54:50,644 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:50,644 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:50,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:50,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:50,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:50,644 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:54:50,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:50,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:50,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:50,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:50,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:50,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:50,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:50,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:54:50,647 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:54:50,647 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:54:50,647 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:54:50,647 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:54:50,647 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:54:50,647 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:54:50,650 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:50,650 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:50,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:50,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:50,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:50,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:50,650 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:50,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:50,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:50,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:50,650 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:50,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:50,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:50,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:50,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:50,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:50,652 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:50,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:50,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:50,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:50,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:50,654 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:50,654 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:50,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:50,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:50,654 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:50,654 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:50,654 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:50,654 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:50,654 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:50,654 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:50,654 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:50,654 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:50,655 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:54:50,655 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:54:50,655 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:54:50,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:50,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:50,655 DEBUG [Thread-1741] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:54:50,655 DEBUG [Thread-1741] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:54:50,673 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:50,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37610 deadline: 1680062150672, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:50,675 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:50,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37572 deadline: 1680062150672, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:50,675 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:50,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37614 deadline: 1680062150673, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:50,677 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:50,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37570 deadline: 1680062150675, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:50,677 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:50,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37634 deadline: 1680062150675, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:50,755 DEBUG [Thread-1741] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:54:50,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:54:50,757 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:54:50,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:54:50,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:54:50,758 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:54:50,758 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:50,758 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:54:50,758 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:54:50,759 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:50,759 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:50,760 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:50,760 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:50,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:50,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:50,760 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:50,760 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:54:50,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:50,761 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:50,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:54:50,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:50,761 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:54:50,761 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:54:50,761 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:54:50,761 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:54:50,761 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:50,761 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:54:50,761 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:54:50,762 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:54:50,762 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:54:50,762 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. started... 2023-03-29 03:54:50,762 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. as already flushing 2023-03-29 03:54:50,762 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:54:50,762 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:54:50,762 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:54:50,762 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:50,762 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:54:50,762 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:54:50,763 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:50,763 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:50,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:50,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:50,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:50,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:50,764 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:50,764 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:54:50,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:50,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:50,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:50,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:50,765 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:54:50,765 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@20f8dd93[Count = 0] remaining members to acquire global barrier 2023-03-29 03:54:50,765 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:54:50,765 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:50,767 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:50,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:50,767 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=167 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/A/20232a5b156349208b03a6a724b74839 2023-03-29 03:54:50,767 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:50,767 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:54:50,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:50,767 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:54:50,768 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:54:50,768 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:54:50,769 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:50,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:50,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:50,769 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:54:50,770 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:50,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:50,770 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:54:50,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:50,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:50,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:50,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:50,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:50,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:50,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:50,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:54:50,774 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:54:50,774 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:54:50,774 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:54:50,774 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:54:50,774 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:54:50,774 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:54:50,776 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:50,776 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:50,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:50,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:50,776 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:50,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:50,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:50,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:50,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:50,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:50,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:50,776 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:50,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:50,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:50,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:50,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:50,778 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:50,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:50,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:50,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:50,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:50,780 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:50,780 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:50,780 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:50,780 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:50,780 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:50,780 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:50,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:50,781 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:54:50,780 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:50,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:50,780 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:50,780 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:54:50,781 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:54:50,781 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:50,781 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:50,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:50,781 DEBUG [Thread-1741] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:54:50,781 DEBUG [Thread-1741] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:54:50,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:50,789 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=167 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/B/7daa29719f94484589b9f3faa4d7640c 2023-03-29 03:54:50,881 DEBUG [Thread-1741] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:54:50,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:54:50,884 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:54:50,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:54:50,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:54:50,885 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:54:50,885 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:50,885 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:54:50,885 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:54:50,886 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:50,886 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:50,891 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:50,891 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:50,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:50,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:50,891 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:50,891 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:54:50,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:50,892 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:50,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:54:50,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:50,892 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:54:50,892 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:54:50,892 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:54:50,893 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:54:50,893 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:50,893 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:54:50,893 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:54:50,893 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:54:50,893 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:54:50,893 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. started... 2023-03-29 03:54:50,893 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. as already flushing 2023-03-29 03:54:50,893 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:54:50,893 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:54:50,893 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:54:50,893 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:50,893 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:54:50,893 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:54:50,897 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:50,897 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:50,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:50,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:50,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:50,897 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:50,897 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:54:50,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:50,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:50,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:50,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:50,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:50,899 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:54:50,899 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1ce9a6fc[Count = 0] remaining members to acquire global barrier 2023-03-29 03:54:50,899 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:54:50,899 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:50,900 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:50,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:50,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:50,900 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:54:50,900 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:54:50,900 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:50,900 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:54:50,900 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:54:50,901 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:54:50,901 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:50,901 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:50,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:50,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:50,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:50,902 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:54:50,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:50,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:50,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:50,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:50,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:50,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:50,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:50,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:54:50,904 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:54:50,904 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:54:50,904 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:54:50,904 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:54:50,904 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:54:50,904 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:54:50,906 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:50,906 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:50,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:50,906 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:50,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:50,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:50,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:50,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:50,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:50,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:50,907 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:50,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:50,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:50,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:50,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:50,908 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:50,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:50,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:50,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:50,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:50,910 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:50,910 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:50,910 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:50,911 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:50,911 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:50,911 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:50,911 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:54:50,910 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:54:50,911 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:54:50,911 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:50,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:50,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:50,911 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:50,911 DEBUG [Thread-1741] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:54:50,911 DEBUG [Thread-1741] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:54:50,911 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:50,911 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:50,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:50,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:50,976 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:50,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37610 deadline: 1680062150975, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:50,976 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:50,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37614 deadline: 1680062150976, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:50,978 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:50,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37570 deadline: 1680062150978, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:50,979 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:50,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37572 deadline: 1680062150978, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:50,981 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:50,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37634 deadline: 1680062150980, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:51,011 DEBUG [Thread-1741] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:54:51,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:54:51,014 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:54:51,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:54:51,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:54:51,015 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:54:51,015 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:51,015 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:54:51,015 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:54:51,015 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:51,015 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:51,016 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:51,016 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:51,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:51,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:51,017 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:51,017 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:54:51,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:51,017 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:51,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:54:51,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:51,018 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:54:51,018 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:54:51,018 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:54:51,018 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:54:51,018 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:51,018 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:54:51,018 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:54:51,018 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:54:51,018 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:54:51,018 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. started... 2023-03-29 03:54:51,018 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. as already flushing 2023-03-29 03:54:51,018 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:54:51,018 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:54:51,018 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:54:51,018 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:51,019 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:54:51,019 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:54:51,020 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:51,020 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:51,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:51,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:51,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:51,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:51,021 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:51,021 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:54:51,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:51,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:51,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:51,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:51,023 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:54:51,023 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2eba386d[Count = 0] remaining members to acquire global barrier 2023-03-29 03:54:51,023 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:54:51,023 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:51,025 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:51,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:51,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:51,025 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:54:51,025 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:51,025 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:54:51,025 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:54:51,025 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:54:51,027 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:54:51,027 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:51,027 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:51,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:51,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:51,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:51,027 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:54:51,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:51,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:51,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:51,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:51,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:51,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:51,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:51,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:54:51,029 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:54:51,029 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:54:51,029 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:54:51,029 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:54:51,029 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:54:51,029 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:54:51,031 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:51,031 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:51,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:51,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:51,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:51,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:51,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:51,031 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:51,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:51,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:51,031 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:51,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:51,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:51,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:51,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:51,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:51,033 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:51,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:51,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:51,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:51,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:51,036 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:51,036 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:51,036 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:51,037 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:51,037 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:51,037 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:51,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:51,037 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:51,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:51,037 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:51,037 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:51,038 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:54:51,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:51,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:51,039 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:51,039 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:54:51,039 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:54:51,039 DEBUG [Thread-1741] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:54:51,039 DEBUG [Thread-1741] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:54:51,139 DEBUG [Thread-1741] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:54:51,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:54:51,141 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:54:51,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:54:51,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:54:51,143 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:54:51,143 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:51,143 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:54:51,143 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:54:51,144 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:51,144 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:51,146 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:51,146 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:51,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:51,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:51,147 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:51,147 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:54:51,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:51,147 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:51,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:54:51,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:51,147 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:54:51,148 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:54:51,148 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:54:51,148 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:54:51,148 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:51,148 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:54:51,148 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:54:51,148 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:54:51,148 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:54:51,148 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. started... 2023-03-29 03:54:51,148 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. as already flushing 2023-03-29 03:54:51,148 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:54:51,148 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:54:51,148 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:54:51,148 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:51,148 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:54:51,148 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:54:51,150 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:51,150 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:51,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:51,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:51,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:51,150 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:51,150 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:54:51,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:51,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:51,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:51,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:51,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:51,152 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:54:51,152 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@53383cc7[Count = 0] remaining members to acquire global barrier 2023-03-29 03:54:51,152 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:54:51,152 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:51,153 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:51,153 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:51,153 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:54:51,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:51,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:51,153 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:54:51,153 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:54:51,153 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:54:51,155 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:51,155 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:54:51,155 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:51,155 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:54:51,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:51,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:51,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:51,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:51,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:51,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:51,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:51,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:51,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:51,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:51,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:54:51,157 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:54:51,157 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:54:51,157 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:54:51,157 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:54:51,157 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:54:51,157 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:54:51,160 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:51,160 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:51,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:51,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:51,160 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:51,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:51,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:51,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:51,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:51,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:51,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:51,161 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:51,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:51,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:51,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:51,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:51,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:51,163 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:51,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:51,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:51,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:51,165 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:51,165 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:51,165 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:51,165 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:51,165 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:54:51,165 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:54:51,165 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:51,165 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:51,165 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:51,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:51,166 DEBUG [Thread-1741] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:54:51,166 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:51,165 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:51,166 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:51,166 DEBUG [Thread-1741] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:54:51,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:51,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:51,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:51,199 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=167 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/C/9762d43c64e540c794ea2ebbbf2c009e 2023-03-29 03:54:51,205 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/A/20232a5b156349208b03a6a724b74839 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/20232a5b156349208b03a6a724b74839 2023-03-29 03:54:51,209 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/20232a5b156349208b03a6a724b74839, entries=200, sequenceid=167, filesize=14.1 K 2023-03-29 03:54:51,210 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/B/7daa29719f94484589b9f3faa4d7640c as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/7daa29719f94484589b9f3faa4d7640c 2023-03-29 03:54:51,214 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/7daa29719f94484589b9f3faa4d7640c, entries=150, sequenceid=167, filesize=11.8 K 2023-03-29 03:54:51,215 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/C/9762d43c64e540c794ea2ebbbf2c009e as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/9762d43c64e540c794ea2ebbbf2c009e 2023-03-29 03:54:51,240 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/9762d43c64e540c794ea2ebbbf2c009e, entries=150, sequenceid=167, filesize=11.8 K 2023-03-29 03:54:51,241 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 2ed55fa255954e1310f6b9fd96e28980 in 883ms, sequenceid=167, compaction requested=true 2023-03-29 03:54:51,241 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2ed55fa255954e1310f6b9fd96e28980: 2023-03-29 03:54:51,242 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-29 03:54:51,242 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:54:51,242 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:54:51,242 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-29 03:54:51,242 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:54:51,243 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 38672 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-29 03:54:51,243 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1912): 2ed55fa255954e1310f6b9fd96e28980/A is initiating minor compaction (all files) 2023-03-29 03:54:51,243 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2ed55fa255954e1310f6b9fd96e28980/A in TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:54:51,243 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/39b6b51e018d43d19d6c3de0f0f27f0d, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/560f1517f0c4405c8668da436fbfeb5a, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/20232a5b156349208b03a6a724b74839] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp, totalSize=37.8 K 2023-03-29 03:54:51,244 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36282 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-29 03:54:51,244 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1912): 2ed55fa255954e1310f6b9fd96e28980/B is initiating minor compaction (all files) 2023-03-29 03:54:51,244 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2ed55fa255954e1310f6b9fd96e28980/B in TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:54:51,244 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/b7e543181c5f490e83467ca868a07b8e, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/71ca5d31c1244abea369822ff32bd33b, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/7daa29719f94484589b9f3faa4d7640c] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp, totalSize=35.4 K 2023-03-29 03:54:51,244 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 39b6b51e018d43d19d6c3de0f0f27f0d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=131, earliestPutTs=1680062086987 2023-03-29 03:54:51,245 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting b7e543181c5f490e83467ca868a07b8e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=131, earliestPutTs=1680062086987 2023-03-29 03:54:51,245 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 560f1517f0c4405c8668da436fbfeb5a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=147, earliestPutTs=1680062088114 2023-03-29 03:54:51,245 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 20232a5b156349208b03a6a724b74839, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=167, earliestPutTs=1680062089241 2023-03-29 03:54:51,245 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 71ca5d31c1244abea369822ff32bd33b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=147, earliestPutTs=1680062088114 2023-03-29 03:54:51,246 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 7daa29719f94484589b9f3faa4d7640c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=167, earliestPutTs=1680062089243 2023-03-29 03:54:51,261 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2ed55fa255954e1310f6b9fd96e28980#A#compaction#253 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:54:51,262 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] throttle.PressureAwareThroughputController(145): 2ed55fa255954e1310f6b9fd96e28980#B#compaction#254 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:54:51,266 DEBUG [Thread-1741] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:54:51,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:54:51,269 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:54:51,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:54:51,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:54:51,271 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:54:51,271 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:51,271 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:54:51,271 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:54:51,272 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:51,272 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:51,273 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:51,273 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:51,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:51,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:51,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:51,274 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:51,274 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:54:51,274 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:51,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:54:51,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:51,275 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:54:51,284 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:54:51,284 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:54:51,284 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:54:51,284 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:51,284 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:54:51,288 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:54:51,288 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:54:51,289 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/B/b608f0638f974340b1bc10205e9b4174 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/b608f0638f974340b1bc10205e9b4174 2023-03-29 03:54:51,289 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:54:51,289 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. started... 2023-03-29 03:54:51,289 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2ed55fa255954e1310f6b9fd96e28980 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-03-29 03:54:51,290 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ed55fa255954e1310f6b9fd96e28980, store=A 2023-03-29 03:54:51,290 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:54:51,291 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ed55fa255954e1310f6b9fd96e28980, store=B 2023-03-29 03:54:51,291 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:54:51,291 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ed55fa255954e1310f6b9fd96e28980, store=C 2023-03-29 03:54:51,291 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:54:51,306 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2ed55fa255954e1310f6b9fd96e28980/B of 2ed55fa255954e1310f6b9fd96e28980 into b608f0638f974340b1bc10205e9b4174(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:54:51,306 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2ed55fa255954e1310f6b9fd96e28980: 2023-03-29 03:54:51,306 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980., storeName=2ed55fa255954e1310f6b9fd96e28980/B, priority=13, startTime=1680062091242; duration=0sec 2023-03-29 03:54:51,306 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:54:51,306 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-29 03:54:51,307 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36282 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-29 03:54:51,307 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1912): 2ed55fa255954e1310f6b9fd96e28980/C is initiating minor compaction (all files) 2023-03-29 03:54:51,307 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2ed55fa255954e1310f6b9fd96e28980/C in TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:54:51,308 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/77de40d9d4b2449ab7e381b4182db937, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/2ac363cfe65a4ad3b584c6745815d5e8, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/9762d43c64e540c794ea2ebbbf2c009e] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp, totalSize=35.4 K 2023-03-29 03:54:51,308 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 77de40d9d4b2449ab7e381b4182db937, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=131, earliestPutTs=1680062086987 2023-03-29 03:54:51,308 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 2ac363cfe65a4ad3b584c6745815d5e8, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=147, earliestPutTs=1680062088114 2023-03-29 03:54:51,309 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 9762d43c64e540c794ea2ebbbf2c009e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=167, earliestPutTs=1680062089243 2023-03-29 03:54:51,321 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/A/bb9be275b7ff43c9ac73f6546f136afa as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/bb9be275b7ff43c9ac73f6546f136afa 2023-03-29 03:54:51,325 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] throttle.PressureAwareThroughputController(145): 2ed55fa255954e1310f6b9fd96e28980#C#compaction#255 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:54:51,327 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2ed55fa255954e1310f6b9fd96e28980/A of 2ed55fa255954e1310f6b9fd96e28980 into bb9be275b7ff43c9ac73f6546f136afa(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:54:51,328 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2ed55fa255954e1310f6b9fd96e28980: 2023-03-29 03:54:51,328 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980., storeName=2ed55fa255954e1310f6b9fd96e28980/A, priority=13, startTime=1680062091241; duration=0sec 2023-03-29 03:54:51,328 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:54:51,354 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/C/3485dbc33ab94d5e9268ad6c63314712 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/3485dbc33ab94d5e9268ad6c63314712 2023-03-29 03:54:51,360 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2ed55fa255954e1310f6b9fd96e28980/C of 2ed55fa255954e1310f6b9fd96e28980 into 3485dbc33ab94d5e9268ad6c63314712(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:54:51,360 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2ed55fa255954e1310f6b9fd96e28980: 2023-03-29 03:54:51,360 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980., storeName=2ed55fa255954e1310f6b9fd96e28980/C, priority=13, startTime=1680062091242; duration=0sec 2023-03-29 03:54:51,360 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:54:51,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on 2ed55fa255954e1310f6b9fd96e28980 2023-03-29 03:54:51,481 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. as already flushing 2023-03-29 03:54:51,501 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:51,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37610 deadline: 1680062151496, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:51,501 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:51,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37634 deadline: 1680062151497, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:51,502 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:51,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37572 deadline: 1680062151498, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:51,503 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:51,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37614 deadline: 1680062151499, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:51,503 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:51,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37570 deadline: 1680062151499, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:51,604 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:51,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37610 deadline: 1680062151602, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:51,604 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:51,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37634 deadline: 1680062151603, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:51,605 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:51,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37572 deadline: 1680062151603, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:51,605 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:51,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37570 deadline: 1680062151605, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:51,607 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:51,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37614 deadline: 1680062151607, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:51,770 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=182 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/A/8dd38d24787c4d5e9ce4d68e3ffb11ec 2023-03-29 03:54:51,808 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=182 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/B/33e41c49ed7c45a98d149b787652e796 2023-03-29 03:54:51,809 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:51,809 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:51,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37570 deadline: 1680062151807, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:51,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37610 deadline: 1680062151808, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:51,809 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:51,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37634 deadline: 1680062151808, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:51,809 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:51,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37572 deadline: 1680062151808, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:51,820 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:51,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37614 deadline: 1680062151818, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:51,851 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=182 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/C/d2c5711d1d274e51b45d4ad197801f4f 2023-03-29 03:54:51,859 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/A/8dd38d24787c4d5e9ce4d68e3ffb11ec as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/8dd38d24787c4d5e9ce4d68e3ffb11ec 2023-03-29 03:54:51,864 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/8dd38d24787c4d5e9ce4d68e3ffb11ec, entries=150, sequenceid=182, filesize=11.8 K 2023-03-29 03:54:51,866 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/B/33e41c49ed7c45a98d149b787652e796 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/33e41c49ed7c45a98d149b787652e796 2023-03-29 03:54:51,869 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/33e41c49ed7c45a98d149b787652e796, entries=150, sequenceid=182, filesize=11.8 K 2023-03-29 03:54:51,870 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/C/d2c5711d1d274e51b45d4ad197801f4f as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/d2c5711d1d274e51b45d4ad197801f4f 2023-03-29 03:54:51,873 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/d2c5711d1d274e51b45d4ad197801f4f, entries=150, sequenceid=182, filesize=11.8 K 2023-03-29 03:54:51,874 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=100.63 KB/103050 for 2ed55fa255954e1310f6b9fd96e28980 in 585ms, sequenceid=182, compaction requested=false 2023-03-29 03:54:51,874 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2ed55fa255954e1310f6b9fd96e28980: 2023-03-29 03:54:51,875 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:54:51,875 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:54:51,875 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:54:51,875 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:51,875 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:54:51,875 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:54:51,877 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:51,877 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:51,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:51,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:51,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:51,877 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:51,877 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:54:51,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:51,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:51,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:51,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:51,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:51,879 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:54:51,879 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@354a6776[Count = 0] remaining members to acquire global barrier 2023-03-29 03:54:51,879 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:54:51,879 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:51,884 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:51,884 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:51,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:51,884 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:54:51,884 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:54:51,884 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:54:51,884 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:51,884 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:54:51,886 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:54:51,886 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:51,886 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:51,886 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:54:51,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:51,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:51,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:51,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:51,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:51,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:51,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:51,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:51,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:51,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:51,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:54:51,888 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:54:51,888 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:54:51,888 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:54:51,888 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:54:51,888 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:54:51,888 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:54:51,890 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:51,890 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:51,890 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:51,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:51,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:51,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:51,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:51,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:51,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:51,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:51,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:51,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:51,891 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:51,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:51,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:51,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:51,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:51,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:51,892 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:51,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:51,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:51,895 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:51,895 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:51,895 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:51,895 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:51,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:51,895 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:51,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:51,895 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:54:51,895 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:54:51,895 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:51,895 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:51,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:51,895 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:51,895 DEBUG [Thread-1741] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:54:51,895 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:51,896 DEBUG [Thread-1741] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:54:51,896 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:51,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:51,996 DEBUG [Thread-1741] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:54:51,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:54:51,998 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:54:51,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:54:51,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:54:51,999 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:54:51,999 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:51,999 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:54:51,999 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:54:51,999 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:51,999 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:52,001 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:52,001 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:52,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:52,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:52,002 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:52,002 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:54:52,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:52,002 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:52,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:54:52,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:52,003 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:54:52,003 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:54:52,003 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:54:52,003 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:54:52,003 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:52,003 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:54:52,003 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:54:52,003 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:54:52,003 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:54:52,003 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. started... 2023-03-29 03:54:52,003 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2ed55fa255954e1310f6b9fd96e28980 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-03-29 03:54:52,004 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ed55fa255954e1310f6b9fd96e28980, store=A 2023-03-29 03:54:52,004 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:54:52,004 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ed55fa255954e1310f6b9fd96e28980, store=B 2023-03-29 03:54:52,004 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:54:52,004 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ed55fa255954e1310f6b9fd96e28980, store=C 2023-03-29 03:54:52,004 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:54:52,016 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=203 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/A/9b264a63081c4a338b515f7228a0e07d 2023-03-29 03:54:52,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on 2ed55fa255954e1310f6b9fd96e28980 2023-03-29 03:54:52,112 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. as already flushing 2023-03-29 03:54:52,124 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:52,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37634 deadline: 1680062152120, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:52,124 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:52,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37610 deadline: 1680062152120, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:52,126 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:52,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37572 deadline: 1680062152124, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:52,127 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:52,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37570 deadline: 1680062152124, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:52,127 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:52,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37614 deadline: 1680062152124, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:52,226 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:52,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37610 deadline: 1680062152225, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:52,226 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:52,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37634 deadline: 1680062152225, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:52,229 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:52,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37572 deadline: 1680062152227, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:52,229 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:52,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37570 deadline: 1680062152228, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:52,429 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:52,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37610 deadline: 1680062152428, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:52,430 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:52,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37634 deadline: 1680062152428, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:52,432 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:52,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37572 deadline: 1680062152430, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:52,432 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:52,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37570 deadline: 1680062152430, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:52,438 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=203 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/B/c46ac230c2cb4b12b740ee7b37621a55 2023-03-29 03:54:52,632 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:52,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37614 deadline: 1680062152632, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:52,733 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:52,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37610 deadline: 1680062152733, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:52,733 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:52,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37572 deadline: 1680062152733, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:52,734 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:52,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37634 deadline: 1680062152733, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:52,736 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:52,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37570 deadline: 1680062152734, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:52,849 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=203 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/C/92128bded558423881b376eacfd1c90e 2023-03-29 03:54:52,859 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/A/9b264a63081c4a338b515f7228a0e07d as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/9b264a63081c4a338b515f7228a0e07d 2023-03-29 03:54:52,862 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/9b264a63081c4a338b515f7228a0e07d, entries=150, sequenceid=203, filesize=11.8 K 2023-03-29 03:54:52,863 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/B/c46ac230c2cb4b12b740ee7b37621a55 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/c46ac230c2cb4b12b740ee7b37621a55 2023-03-29 03:54:52,867 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/c46ac230c2cb4b12b740ee7b37621a55, entries=150, sequenceid=203, filesize=11.8 K 2023-03-29 03:54:52,867 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/C/92128bded558423881b376eacfd1c90e as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/92128bded558423881b376eacfd1c90e 2023-03-29 03:54:52,871 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/92128bded558423881b376eacfd1c90e, entries=150, sequenceid=203, filesize=11.8 K 2023-03-29 03:54:52,871 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=80.51 KB/82440 for 2ed55fa255954e1310f6b9fd96e28980 in 868ms, sequenceid=203, compaction requested=true 2023-03-29 03:54:52,872 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2ed55fa255954e1310f6b9fd96e28980: 2023-03-29 03:54:52,872 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:54:52,872 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:54:52,872 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:54:52,872 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:52,872 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:54:52,872 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:54:52,873 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:52,873 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:52,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:52,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:52,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:52,874 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:52,874 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:54:52,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:52,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:52,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:52,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:52,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:52,876 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:54:52,876 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6445f965[Count = 0] remaining members to acquire global barrier 2023-03-29 03:54:52,876 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:54:52,876 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:52,877 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:52,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:52,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:52,877 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:54:52,877 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:54:52,877 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:54:52,877 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:52,877 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:54:52,878 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:54:52,878 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:52,879 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:52,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:52,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:52,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:52,879 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:54:52,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:52,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:52,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:52,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:52,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:52,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:52,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:52,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:54:52,882 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:54:52,882 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:54:52,882 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:54:52,882 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:54:52,882 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:54:52,882 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:54:52,885 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:52,885 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:52,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:52,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:52,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:52,885 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:52,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:52,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:52,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:52,885 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:52,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:52,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:52,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:52,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:52,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:52,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:52,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:52,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:52,887 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:52,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:52,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:52,889 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:52,889 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:52,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:52,889 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:52,889 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:54:52,889 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:54:52,889 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:52,889 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:52,889 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:52,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:52,890 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:52,890 DEBUG [Thread-1741] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:54:52,890 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:52,890 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:52,890 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:52,890 DEBUG [Thread-1741] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:54:52,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:52,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:52,990 DEBUG [Thread-1741] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:54:52,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:54:52,992 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:54:52,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:54:52,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:54:52,996 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:54:52,996 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:52,996 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:54:52,996 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:54:52,997 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:52,997 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:52,998 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:52,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:52,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:52,998 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:52,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:52,999 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:52,999 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:52,999 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:54:52,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:54:52,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:52,999 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:54:53,000 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:54:53,000 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:54:53,000 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:54:53,000 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:53,000 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:54:53,000 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:54:53,000 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:54:53,000 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:54:53,000 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. started... 2023-03-29 03:54:53,000 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2ed55fa255954e1310f6b9fd96e28980 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-03-29 03:54:53,001 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ed55fa255954e1310f6b9fd96e28980, store=A 2023-03-29 03:54:53,001 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:54:53,001 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ed55fa255954e1310f6b9fd96e28980, store=B 2023-03-29 03:54:53,001 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:54:53,001 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ed55fa255954e1310f6b9fd96e28980, store=C 2023-03-29 03:54:53,001 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:54:53,011 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=218 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/A/541188092080413aaaf5a9585a5d57c3 2023-03-29 03:54:53,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on 2ed55fa255954e1310f6b9fd96e28980 2023-03-29 03:54:53,240 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. as already flushing 2023-03-29 03:54:53,256 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:53,256 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:53,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37572 deadline: 1680062153250, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:53,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37634 deadline: 1680062153250, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:53,260 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:53,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37610 deadline: 1680062153256, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:53,260 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:53,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37570 deadline: 1680062153256, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:53,358 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:53,358 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:53,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37572 deadline: 1680062153357, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:53,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37634 deadline: 1680062153357, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:53,362 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:53,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37610 deadline: 1680062153361, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:53,362 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:53,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37570 deadline: 1680062153361, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:53,426 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=218 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/B/054a5dcdbed044a8b6dedb4201f32c08 2023-03-29 03:54:53,561 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:53,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37634 deadline: 1680062153560, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:53,562 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:53,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37572 deadline: 1680062153560, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:53,565 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:53,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37610 deadline: 1680062153564, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:53,565 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:53,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37570 deadline: 1680062153564, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:53,640 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:53,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37614 deadline: 1680062153638, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:53,835 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=218 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/C/abe79ae3798b48d9a58fbc0c99178013 2023-03-29 03:54:53,840 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/A/541188092080413aaaf5a9585a5d57c3 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/541188092080413aaaf5a9585a5d57c3 2023-03-29 03:54:53,844 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/541188092080413aaaf5a9585a5d57c3, entries=100, sequenceid=218, filesize=9.4 K 2023-03-29 03:54:53,845 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/B/054a5dcdbed044a8b6dedb4201f32c08 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/054a5dcdbed044a8b6dedb4201f32c08 2023-03-29 03:54:53,849 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/054a5dcdbed044a8b6dedb4201f32c08, entries=100, sequenceid=218, filesize=9.4 K 2023-03-29 03:54:53,849 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/C/abe79ae3798b48d9a58fbc0c99178013 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/abe79ae3798b48d9a58fbc0c99178013 2023-03-29 03:54:53,852 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/abe79ae3798b48d9a58fbc0c99178013, entries=100, sequenceid=218, filesize=9.4 K 2023-03-29 03:54:53,853 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 2ed55fa255954e1310f6b9fd96e28980 in 853ms, sequenceid=218, compaction requested=true 2023-03-29 03:54:53,853 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2ed55fa255954e1310f6b9fd96e28980: 2023-03-29 03:54:53,853 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:54:53,853 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:54:53,853 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:54:53,853 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:53,854 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:54:53,854 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:54:53,855 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:53,855 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:53,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:53,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:53,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:53,856 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:53,856 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:54:53,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:53,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:53,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:53,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:53,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:53,857 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:54:53,857 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1bc3ccd0[Count = 0] remaining members to acquire global barrier 2023-03-29 03:54:53,857 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:54:53,857 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:53,859 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:53,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:53,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:53,859 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:54:53,859 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:54:53,859 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:53,859 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:54:53,859 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:54:53,860 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:54:53,860 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:53,860 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:53,861 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:54:53,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:53,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:53,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:53,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:53,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:53,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:53,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:53,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:53,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:53,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:53,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:54:53,863 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:54:53,863 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:54:53,863 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:54:53,863 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:54:53,863 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:54:53,863 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:54:53,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on 2ed55fa255954e1310f6b9fd96e28980 2023-03-29 03:54:53,865 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2ed55fa255954e1310f6b9fd96e28980 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-03-29 03:54:53,865 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ed55fa255954e1310f6b9fd96e28980, store=A 2023-03-29 03:54:53,865 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:54:53,865 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ed55fa255954e1310f6b9fd96e28980, store=B 2023-03-29 03:54:53,865 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:54:53,865 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ed55fa255954e1310f6b9fd96e28980, store=C 2023-03-29 03:54:53,865 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:54:53,865 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:53,865 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:53,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:53,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:53,865 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:53,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:53,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:53,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:53,866 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:53,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:53,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:53,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:53,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:53,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:53,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:53,867 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:53,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:53,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:53,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:53,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:53,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:53,870 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:53,870 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:53,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:53,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:53,871 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:53,871 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:53,871 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:53,871 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:53,871 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:54:53,871 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:54:53,871 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:53,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:53,871 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:53,871 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:53,871 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:53,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:53,871 DEBUG [Thread-1741] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:54:53,871 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:54:53,871 DEBUG [Thread-1741] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:54:53,877 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:53,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37572 deadline: 1680062153875, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:53,878 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:53,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37570 deadline: 1680062153875, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:53,882 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:53,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37634 deadline: 1680062153878, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:53,882 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:53,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37610 deadline: 1680062153878, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:53,971 DEBUG [Thread-1741] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:54:53,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:54:53,973 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:54:53,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:54:53,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:54:53,975 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:54:53,975 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:53,975 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:54:53,975 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:54:53,975 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:53,975 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:53,977 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:53,977 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:53,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:53,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:53,978 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:53,978 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:54:53,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:53,978 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:53,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:54:53,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:53,979 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:54:53,980 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:54:53,980 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:54:53,980 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:53,980 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:54:53,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37572 deadline: 1680062153978, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:53,980 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:53,980 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:54:53,980 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:53,980 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:54:53,980 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:54:53,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37570 deadline: 1680062153979, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:53,980 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:54:53,980 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. started... 2023-03-29 03:54:53,980 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. as already flushing 2023-03-29 03:54:53,980 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:54:53,980 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:54:53,981 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:54:53,981 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:53,981 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:54:53,981 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:54:53,982 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:53,982 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:53,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:53,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:53,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:53,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:53,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:53,983 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:53,983 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:54:53,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:53,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:53,984 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:53,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37634 deadline: 1680062153983, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:53,984 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:53,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37610 deadline: 1680062153983, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:53,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:53,985 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:54:53,985 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@98f156b[Count = 0] remaining members to acquire global barrier 2023-03-29 03:54:53,985 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:54:53,985 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:53,986 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:53,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:53,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:53,986 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:54:53,986 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:54:53,986 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:54:53,987 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:53,987 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:54:53,991 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:53,991 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:54:53,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:53,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:53,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:53,991 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:53,991 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:54:53,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:53,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:53,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:53,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:53,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:53,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:53,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:53,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:54:53,994 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:54:53,994 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:54:53,994 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:54:53,994 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:54:53,994 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:54:53,994 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:54:53,995 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:53,995 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:53,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:53,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:53,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:53,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:53,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:53,996 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:53,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:53,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:53,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:53,996 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:53,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:53,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:53,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:53,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:53,997 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:53,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:53,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:53,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:53,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:53,999 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:53,999 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:53,999 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:54,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:54,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:54,000 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:54:54,000 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:54:54,000 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:54,000 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:54,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:54,000 DEBUG [Thread-1741] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:54:54,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:54,000 DEBUG [Thread-1741] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:54:54,000 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:54,000 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:54,000 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:54,000 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:54,000 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:54,100 DEBUG [Thread-1741] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:54:54,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:54:54,105 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:54:54,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:54:54,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:54:54,107 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:54:54,107 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:54,107 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:54:54,107 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:54:54,107 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:54,108 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:54,109 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:54,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:54,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:54,111 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:54,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:54,111 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:54,111 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:54:54,111 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:54,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:54:54,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:54,112 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:54:54,112 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:54:54,112 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:54:54,112 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:54:54,112 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:54,112 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:54:54,120 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:54:54,120 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:54:54,124 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:54:54,124 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. started... 2023-03-29 03:54:54,124 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. as already flushing 2023-03-29 03:54:54,124 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:54:54,124 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:54:54,124 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:54:54,124 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:54,124 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:54:54,124 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:54:54,126 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:54,126 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:54,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:54,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:54,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:54,126 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:54,126 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:54:54,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:54,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:54,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:54,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:54,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:54,128 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:54:54,128 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f98a2fb[Count = 0] remaining members to acquire global barrier 2023-03-29 03:54:54,128 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:54:54,128 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:54,129 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:54,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:54,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:54,130 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:54:54,130 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:54,130 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:54:54,130 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:54:54,130 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:54:54,132 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:54,132 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:54:54,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:54,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:54,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:54,132 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:54,133 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:54:54,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:54,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:54,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:54,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:54,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:54,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:54,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:54,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:54:54,135 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:54:54,135 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:54:54,135 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:54:54,135 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:54:54,135 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:54:54,135 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:54:54,137 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:54,137 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:54,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:54,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:54,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:54,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:54,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:54,137 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:54,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:54,137 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:54,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:54,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:54,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:54,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:54,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:54,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:54,138 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:54,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:54,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:54,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:54,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:54,141 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:54,141 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:54,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:54,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:54,141 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:54,141 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:54,141 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:54,141 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:54,141 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:54,141 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:54,141 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:54,141 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:54:54,141 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:54:54,142 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:54:54,141 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:54,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:54,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:54,142 DEBUG [Thread-1741] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:54:54,142 DEBUG [Thread-1741] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:54:54,183 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:54,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37572 deadline: 1680062154181, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:54,183 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:54,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37570 deadline: 1680062154181, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:54,188 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:54,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37634 deadline: 1680062154186, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:54,188 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:54,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37610 deadline: 1680062154186, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:54,242 DEBUG [Thread-1741] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:54:54,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:54:54,244 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:54:54,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:54:54,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:54:54,246 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:54:54,246 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:54,246 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:54:54,246 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:54:54,246 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:54,246 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:54,247 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:54,247 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:54,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:54,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:54,248 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:54,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:54,248 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:54:54,248 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:54,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:54:54,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:54,249 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:54:54,249 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:54:54,249 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:54:54,249 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:54:54,249 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:54,249 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:54:54,249 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:54:54,249 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:54:54,249 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:54:54,249 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. started... 2023-03-29 03:54:54,249 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. as already flushing 2023-03-29 03:54:54,249 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:54:54,249 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:54:54,250 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:54:54,250 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:54,250 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:54:54,250 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:54:54,252 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:54,252 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:54,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:54,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:54,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:54,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:54,253 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:54,253 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:54:54,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:54,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:54,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:54,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:54,254 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:54:54,254 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@24dde0bc[Count = 0] remaining members to acquire global barrier 2023-03-29 03:54:54,254 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:54:54,254 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:54,255 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:54,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:54,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:54,255 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:54:54,255 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:54:54,255 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:54,255 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:54:54,255 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:54:54,256 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:54:54,257 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:54,256 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:54,257 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:54:54,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:54,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:54,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:54,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:54,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:54,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:54,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:54,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:54,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:54,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:54,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:54:54,259 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:54:54,259 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:54:54,259 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:54:54,259 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:54:54,259 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:54:54,259 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:54:54,261 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:54,261 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:54,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:54,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:54,261 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:54,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:54,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:54,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:54,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:54,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:54,262 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:54,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:54,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:54,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:54,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:54,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:54,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:54,263 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:54,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:54,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:54,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:54,266 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:54,266 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:54,266 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:54,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:54,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:54,266 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:54,267 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:54,266 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:54,267 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:54:54,267 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:54:54,267 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:54,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:54,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:54,267 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:54,267 DEBUG [Thread-1741] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:54:54,267 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:54,267 DEBUG [Thread-1741] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:54:54,267 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:54,273 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=236 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/A/bdc9d545c90f4638ad8bb52924252650 2023-03-29 03:54:54,367 DEBUG [Thread-1741] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:54:54,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:54:54,369 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:54:54,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:54:54,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:54:54,371 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:54:54,371 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:54,371 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:54:54,371 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:54:54,371 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:54,371 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:54,373 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:54,373 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:54,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:54,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:54,373 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:54,373 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:54:54,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:54,373 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:54,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:54:54,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:54,374 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:54:54,374 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:54:54,374 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:54:54,374 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:54:54,374 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:54,374 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:54:54,374 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:54:54,374 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:54:54,375 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:54:54,375 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. started... 2023-03-29 03:54:54,375 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. as already flushing 2023-03-29 03:54:54,375 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:54:54,375 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:54:54,375 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:54:54,375 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:54,375 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:54:54,375 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:54:54,376 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:54,376 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:54,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:54,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:54,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:54,377 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:54,377 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:54:54,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:54,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:54,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:54,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:54,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:54,378 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:54:54,378 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:54:54,378 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@217d0dbb[Count = 0] remaining members to acquire global barrier 2023-03-29 03:54:54,378 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:54,379 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:54,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:54,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:54,379 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:54:54,379 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:54:54,379 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:54:54,380 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:54,380 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:54:54,382 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:54,382 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:54:54,382 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:54,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:54,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:54,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:54,382 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:54:54,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:54,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:54,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:54,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:54,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:54,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:54,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:54,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:54:54,385 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:54:54,385 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:54:54,385 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:54:54,385 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:54:54,385 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:54:54,385 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:54:54,387 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:54,387 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:54,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:54,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:54,387 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:54,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:54,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:54,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:54,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:54,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:54,388 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:54,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:54,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:54,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:54,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:54,389 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:54,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:54,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:54,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:54,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:54,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:54,394 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:54,394 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:54,394 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:54,394 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:54,394 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:54,394 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:54,394 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:54,394 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:54:54,394 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:54:54,394 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:54,394 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:54,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:54,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:54,395 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:54:54,394 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:54,395 DEBUG [Thread-1741] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:54:54,395 DEBUG [Thread-1741] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:54:54,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:54,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:54,486 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:54,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37572 deadline: 1680062154484, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:54,486 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:54,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37570 deadline: 1680062154484, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:54,492 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:54,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37610 deadline: 1680062154490, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:54,493 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:54,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37634 deadline: 1680062154491, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:54,495 DEBUG [Thread-1741] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:54:54,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:54:54,497 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:54:54,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:54:54,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:54:54,508 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:54:54,508 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:54,508 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:54:54,508 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:54:54,509 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:54,509 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:54,511 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:54,511 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:54,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:54,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:54,511 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:54,511 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:54:54,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:54,512 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:54,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:54:54,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:54,512 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:54:54,512 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:54:54,512 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:54:54,513 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:54:54,513 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:54,513 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:54:54,513 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:54:54,513 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:54:54,513 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:54:54,513 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. started... 2023-03-29 03:54:54,513 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. as already flushing 2023-03-29 03:54:54,513 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:54:54,513 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:54:54,513 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:54:54,513 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:54,513 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:54:54,513 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:54:54,515 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:54,515 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:54,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:54,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:54,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:54,515 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:54,516 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:54:54,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:54,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:54,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:54,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:54,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:54,517 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:54:54,517 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@50a860b2[Count = 0] remaining members to acquire global barrier 2023-03-29 03:54:54,517 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:54:54,517 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:54,518 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:54,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:54,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:54,519 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:54:54,519 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:54:54,519 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:54,519 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:54:54,519 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:54:54,520 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:54,520 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:54:54,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:54,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:54,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:54,520 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:54,520 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:54:54,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:54,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:54,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:54,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:54,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:54,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:54,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:54,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:54:54,523 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:54:54,523 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:54:54,523 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:54:54,523 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:54:54,523 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:54:54,523 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:54:54,526 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:54,526 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:54,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:54,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:54,526 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:54,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:54,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:54,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:54,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:54,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:54,527 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:54,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:54,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:54,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:54,529 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:54,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:54,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:54,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:54,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:54,532 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:54,532 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:54,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:54,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:54,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:54,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:54,536 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:54,536 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:54,537 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:54,537 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:54,536 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:54:54,537 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:54:54,537 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:54,537 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:54,537 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:54,537 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:54,538 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:54:54,538 DEBUG [Thread-1741] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:54:54,538 DEBUG [Thread-1741] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:54:54,638 DEBUG [Thread-1741] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:54:54,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:54:54,640 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:54:54,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:54:54,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:54:54,641 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:54:54,641 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:54,641 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:54:54,641 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:54:54,642 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:54,642 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:54,643 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:54,643 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:54,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:54,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:54,644 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:54,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:54,644 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:54:54,644 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:54,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:54:54,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:54,644 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:54:54,645 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:54:54,645 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:54:54,645 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:54:54,645 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:54,645 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:54:54,645 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:54:54,645 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:54:54,645 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:54:54,645 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. started... 2023-03-29 03:54:54,645 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. as already flushing 2023-03-29 03:54:54,645 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:54:54,645 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:54:54,645 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:54:54,645 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:54,645 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:54:54,645 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:54:54,647 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:54,647 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:54,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:54,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:54,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:54,647 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:54,647 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:54:54,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:54,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:54,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:54,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:54,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:54,648 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:54:54,648 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a0d8f26[Count = 0] remaining members to acquire global barrier 2023-03-29 03:54:54,648 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:54:54,649 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:54,653 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:54,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:54,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:54,653 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:54:54,653 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:54:54,653 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:54:54,653 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:54,653 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:54:54,655 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:54,655 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:54:54,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:54,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:54,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:54,655 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:54,655 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:54:54,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:54,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:54,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:54,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:54,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:54,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:54,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:54,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:54:54,657 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:54:54,657 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:54:54,657 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:54:54,657 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:54:54,657 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:54:54,657 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:54:54,659 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:54,659 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:54,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:54,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:54,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:54,659 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:54,659 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:54,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:54,659 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:54,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:54,659 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:54,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:54,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:54,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:54,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:54,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:54,661 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:54,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:54,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:54,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:54,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:54,663 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:54,663 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:54,663 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:54,663 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:54:54,663 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:54:54,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:54,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:54,663 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:54,663 DEBUG [Thread-1741] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:54:54,663 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:54,664 DEBUG [Thread-1741] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:54:54,664 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:54:54,664 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:54,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:54,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:54,664 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:54,664 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:54,664 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:54,664 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:54,683 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=236 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/B/174ddd6b9710415480e0e1c0a2a0f76a 2023-03-29 03:54:54,718 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=236 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/C/0f5f678ea4964ffe857b2b58b01d2c8e 2023-03-29 03:54:54,723 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/A/bdc9d545c90f4638ad8bb52924252650 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/bdc9d545c90f4638ad8bb52924252650 2023-03-29 03:54:54,728 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/bdc9d545c90f4638ad8bb52924252650, entries=150, sequenceid=236, filesize=11.8 K 2023-03-29 03:54:54,729 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/B/174ddd6b9710415480e0e1c0a2a0f76a as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/174ddd6b9710415480e0e1c0a2a0f76a 2023-03-29 03:54:54,733 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/174ddd6b9710415480e0e1c0a2a0f76a, entries=150, sequenceid=236, filesize=11.8 K 2023-03-29 03:54:54,733 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/C/0f5f678ea4964ffe857b2b58b01d2c8e as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/0f5f678ea4964ffe857b2b58b01d2c8e 2023-03-29 03:54:54,750 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/0f5f678ea4964ffe857b2b58b01d2c8e, entries=150, sequenceid=236, filesize=11.8 K 2023-03-29 03:54:54,751 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 2ed55fa255954e1310f6b9fd96e28980 in 887ms, sequenceid=236, compaction requested=true 2023-03-29 03:54:54,751 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2ed55fa255954e1310f6b9fd96e28980: 2023-03-29 03:54:54,751 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:54:54,751 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-29 03:54:54,751 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-29 03:54:54,751 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:54:54,751 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:54:54,753 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 58162 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-29 03:54:54,753 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1912): 2ed55fa255954e1310f6b9fd96e28980/A is initiating minor compaction (all files) 2023-03-29 03:54:54,753 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 58162 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-29 03:54:54,753 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2ed55fa255954e1310f6b9fd96e28980/A in TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:54:54,753 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1912): 2ed55fa255954e1310f6b9fd96e28980/B is initiating minor compaction (all files) 2023-03-29 03:54:54,753 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/bb9be275b7ff43c9ac73f6546f136afa, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/8dd38d24787c4d5e9ce4d68e3ffb11ec, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/9b264a63081c4a338b515f7228a0e07d, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/541188092080413aaaf5a9585a5d57c3, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/bdc9d545c90f4638ad8bb52924252650] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp, totalSize=56.8 K 2023-03-29 03:54:54,753 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2ed55fa255954e1310f6b9fd96e28980/B in TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:54:54,753 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/b608f0638f974340b1bc10205e9b4174, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/33e41c49ed7c45a98d149b787652e796, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/c46ac230c2cb4b12b740ee7b37621a55, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/054a5dcdbed044a8b6dedb4201f32c08, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/174ddd6b9710415480e0e1c0a2a0f76a] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp, totalSize=56.8 K 2023-03-29 03:54:54,753 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting bb9be275b7ff43c9ac73f6546f136afa, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=167, earliestPutTs=1680062089243 2023-03-29 03:54:54,753 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting b608f0638f974340b1bc10205e9b4174, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=167, earliestPutTs=1680062089243 2023-03-29 03:54:54,754 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 8dd38d24787c4d5e9ce4d68e3ffb11ec, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=182, earliestPutTs=1680062090365 2023-03-29 03:54:54,754 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 33e41c49ed7c45a98d149b787652e796, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=182, earliestPutTs=1680062090365 2023-03-29 03:54:54,754 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 9b264a63081c4a338b515f7228a0e07d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=203, earliestPutTs=1680062091494 2023-03-29 03:54:54,754 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting c46ac230c2cb4b12b740ee7b37621a55, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=203, earliestPutTs=1680062091494 2023-03-29 03:54:54,754 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 541188092080413aaaf5a9585a5d57c3, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=218, earliestPutTs=1680062092119 2023-03-29 03:54:54,754 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 054a5dcdbed044a8b6dedb4201f32c08, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=218, earliestPutTs=1680062092119 2023-03-29 03:54:54,755 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting bdc9d545c90f4638ad8bb52924252650, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=236, earliestPutTs=1680062093244 2023-03-29 03:54:54,755 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 174ddd6b9710415480e0e1c0a2a0f76a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=236, earliestPutTs=1680062093244 2023-03-29 03:54:54,764 DEBUG [Thread-1741] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:54:54,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:54:54,766 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:54:54,766 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2ed55fa255954e1310f6b9fd96e28980#A#compaction#269 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:54:54,766 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] throttle.PressureAwareThroughputController(145): 2ed55fa255954e1310f6b9fd96e28980#B#compaction#268 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:54:54,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:54:54,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:54:54,767 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:54:54,767 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:54,767 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:54:54,767 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:54:54,768 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:54,768 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:54,771 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:54,771 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:54,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:54,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:54,771 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:54,771 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:54:54,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:54,772 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:54,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:54:54,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:54,772 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:54:54,796 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:54:54,796 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:54:54,796 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:54:54,796 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:54,796 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:54:54,824 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:54:54,824 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:54:54,828 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:54:54,828 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. started... 2023-03-29 03:54:54,829 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2ed55fa255954e1310f6b9fd96e28980 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-03-29 03:54:54,829 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ed55fa255954e1310f6b9fd96e28980, store=A 2023-03-29 03:54:54,830 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:54:54,830 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ed55fa255954e1310f6b9fd96e28980, store=B 2023-03-29 03:54:54,830 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:54:54,830 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ed55fa255954e1310f6b9fd96e28980, store=C 2023-03-29 03:54:54,830 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:54:54,850 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/A/5a2f8b2971fd4ceb99d780f2ff43116e as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/5a2f8b2971fd4ceb99d780f2ff43116e 2023-03-29 03:54:54,855 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/B/9e0a1699f9294c21a2c3bc9346973d84 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/9e0a1699f9294c21a2c3bc9346973d84 2023-03-29 03:54:54,863 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 2ed55fa255954e1310f6b9fd96e28980/A of 2ed55fa255954e1310f6b9fd96e28980 into 5a2f8b2971fd4ceb99d780f2ff43116e(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:54:54,863 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2ed55fa255954e1310f6b9fd96e28980: 2023-03-29 03:54:54,863 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980., storeName=2ed55fa255954e1310f6b9fd96e28980/A, priority=11, startTime=1680062094751; duration=0sec 2023-03-29 03:54:54,863 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:54:54,863 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-29 03:54:54,865 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 58162 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-29 03:54:54,865 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 2ed55fa255954e1310f6b9fd96e28980/B of 2ed55fa255954e1310f6b9fd96e28980 into 9e0a1699f9294c21a2c3bc9346973d84(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:54:54,865 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1912): 2ed55fa255954e1310f6b9fd96e28980/C is initiating minor compaction (all files) 2023-03-29 03:54:54,865 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2ed55fa255954e1310f6b9fd96e28980: 2023-03-29 03:54:54,865 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980., storeName=2ed55fa255954e1310f6b9fd96e28980/B, priority=11, startTime=1680062094751; duration=0sec 2023-03-29 03:54:54,865 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2ed55fa255954e1310f6b9fd96e28980/C in TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:54:54,865 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:54:54,865 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/3485dbc33ab94d5e9268ad6c63314712, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/d2c5711d1d274e51b45d4ad197801f4f, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/92128bded558423881b376eacfd1c90e, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/abe79ae3798b48d9a58fbc0c99178013, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/0f5f678ea4964ffe857b2b58b01d2c8e] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp, totalSize=56.8 K 2023-03-29 03:54:54,866 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 3485dbc33ab94d5e9268ad6c63314712, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=167, earliestPutTs=1680062089243 2023-03-29 03:54:54,866 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting d2c5711d1d274e51b45d4ad197801f4f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=182, earliestPutTs=1680062090365 2023-03-29 03:54:54,866 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 92128bded558423881b376eacfd1c90e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=203, earliestPutTs=1680062091494 2023-03-29 03:54:54,866 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting abe79ae3798b48d9a58fbc0c99178013, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=218, earliestPutTs=1680062092119 2023-03-29 03:54:54,867 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 0f5f678ea4964ffe857b2b58b01d2c8e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=236, earliestPutTs=1680062093244 2023-03-29 03:54:54,870 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=250 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/A/4d20d8dcbd8a4f0e808fe0b86874bcff 2023-03-29 03:54:54,886 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2ed55fa255954e1310f6b9fd96e28980#C#compaction#271 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:54:54,912 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=250 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/B/f11c9143d9b541f5b7526fe38cb30d41 2023-03-29 03:54:54,922 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/C/da7a8b072069472e9a74756697f88520 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/da7a8b072069472e9a74756697f88520 2023-03-29 03:54:54,928 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 2ed55fa255954e1310f6b9fd96e28980/C of 2ed55fa255954e1310f6b9fd96e28980 into da7a8b072069472e9a74756697f88520(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:54:54,928 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2ed55fa255954e1310f6b9fd96e28980: 2023-03-29 03:54:54,928 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980., storeName=2ed55fa255954e1310f6b9fd96e28980/C, priority=11, startTime=1680062094751; duration=0sec 2023-03-29 03:54:54,928 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:54:54,944 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=250 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/C/977142afe896430c8962ea3bdf5f054b 2023-03-29 03:54:54,949 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/A/4d20d8dcbd8a4f0e808fe0b86874bcff as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/4d20d8dcbd8a4f0e808fe0b86874bcff 2023-03-29 03:54:54,954 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/4d20d8dcbd8a4f0e808fe0b86874bcff, entries=150, sequenceid=250, filesize=11.8 K 2023-03-29 03:54:54,955 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/B/f11c9143d9b541f5b7526fe38cb30d41 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/f11c9143d9b541f5b7526fe38cb30d41 2023-03-29 03:54:54,958 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/f11c9143d9b541f5b7526fe38cb30d41, entries=150, sequenceid=250, filesize=11.8 K 2023-03-29 03:54:54,960 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/C/977142afe896430c8962ea3bdf5f054b as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/977142afe896430c8962ea3bdf5f054b 2023-03-29 03:54:54,963 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/977142afe896430c8962ea3bdf5f054b, entries=150, sequenceid=250, filesize=11.8 K 2023-03-29 03:54:54,964 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=0 B/0 for 2ed55fa255954e1310f6b9fd96e28980 in 135ms, sequenceid=250, compaction requested=false 2023-03-29 03:54:54,965 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2ed55fa255954e1310f6b9fd96e28980: 2023-03-29 03:54:54,965 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:54:54,965 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:54:54,965 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:54:54,965 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:54,965 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:54:54,965 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:54:54,966 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:54,967 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:54,967 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:54:54,966 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:54,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:54,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:54,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:54,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:54,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:54,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:54,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:54,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:54,969 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:54:54,969 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@311e48db[Count = 0] remaining members to acquire global barrier 2023-03-29 03:54:54,969 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:54:54,969 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:54,971 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:54,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:54,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:54,971 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:54:54,971 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:54:54,971 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:54:54,971 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:54,971 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:54:54,973 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:54,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:54,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:54,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:54,973 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:54:54,973 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:54,973 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:54:54,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:54,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:54,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:54,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:54,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:54,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:54,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:54,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:54:54,976 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:54:54,976 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:54:54,976 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:54:54,976 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:54:54,976 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:54:54,976 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:54:54,979 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:54,979 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:54,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:54,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:54,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:54,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:54,979 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:54,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:54,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:54,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:54,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:54,980 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:54,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:54,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:54,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:54,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:54,981 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:54,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:54,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:54,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:54,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:54,986 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:54,986 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:54,986 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:54,986 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:54:54,986 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:54:54,986 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:54,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:54,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:54,986 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:54,986 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:54,986 DEBUG [Thread-1741] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:54:54,986 DEBUG [Thread-1741] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:54:54,986 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:54,986 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:54:54,986 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:54,986 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:54,986 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:54,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:54,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:54,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on 2ed55fa255954e1310f6b9fd96e28980 2023-03-29 03:54:54,997 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2ed55fa255954e1310f6b9fd96e28980 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-29 03:54:54,998 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ed55fa255954e1310f6b9fd96e28980, store=A 2023-03-29 03:54:54,998 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:54:54,998 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ed55fa255954e1310f6b9fd96e28980, store=B 2023-03-29 03:54:54,998 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:54:54,998 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ed55fa255954e1310f6b9fd96e28980, store=C 2023-03-29 03:54:54,998 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:54:55,007 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=265 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/A/e5a831d240cc4946ae03c60f2b9a9cf9 2023-03-29 03:54:55,018 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:55,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37572 deadline: 1680062155015, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:55,019 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:55,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37634 deadline: 1680062155017, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:55,019 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:55,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37610 deadline: 1680062155017, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:55,021 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:55,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37570 deadline: 1680062155018, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:55,021 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=265 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/B/e40baabca095450f9e91c9ee6137a521 2023-03-29 03:54:55,031 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=265 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/C/20059f61c4214d0891e5653b6d4dbd17 2023-03-29 03:54:55,035 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/A/e5a831d240cc4946ae03c60f2b9a9cf9 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/e5a831d240cc4946ae03c60f2b9a9cf9 2023-03-29 03:54:55,038 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/e5a831d240cc4946ae03c60f2b9a9cf9, entries=100, sequenceid=265, filesize=9.5 K 2023-03-29 03:54:55,039 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/B/e40baabca095450f9e91c9ee6137a521 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/e40baabca095450f9e91c9ee6137a521 2023-03-29 03:54:55,042 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/e40baabca095450f9e91c9ee6137a521, entries=100, sequenceid=265, filesize=9.5 K 2023-03-29 03:54:55,043 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/C/20059f61c4214d0891e5653b6d4dbd17 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/20059f61c4214d0891e5653b6d4dbd17 2023-03-29 03:54:55,057 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/20059f61c4214d0891e5653b6d4dbd17, entries=100, sequenceid=265, filesize=9.5 K 2023-03-29 03:54:55,058 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 2ed55fa255954e1310f6b9fd96e28980 in 61ms, sequenceid=265, compaction requested=true 2023-03-29 03:54:55,058 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2ed55fa255954e1310f6b9fd96e28980: 2023-03-29 03:54:55,059 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:54:55,059 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-29 03:54:55,059 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:54:55,059 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-29 03:54:55,059 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:54:55,060 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34360 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-29 03:54:55,060 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34360 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-29 03:54:55,061 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1912): 2ed55fa255954e1310f6b9fd96e28980/B is initiating minor compaction (all files) 2023-03-29 03:54:55,061 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2ed55fa255954e1310f6b9fd96e28980/B in TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:54:55,061 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/9e0a1699f9294c21a2c3bc9346973d84, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/f11c9143d9b541f5b7526fe38cb30d41, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/e40baabca095450f9e91c9ee6137a521] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp, totalSize=33.6 K 2023-03-29 03:54:55,061 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 9e0a1699f9294c21a2c3bc9346973d84, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=236, earliestPutTs=1680062093244 2023-03-29 03:54:55,062 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1912): 2ed55fa255954e1310f6b9fd96e28980/A is initiating minor compaction (all files) 2023-03-29 03:54:55,062 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2ed55fa255954e1310f6b9fd96e28980/A in TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:54:55,062 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/5a2f8b2971fd4ceb99d780f2ff43116e, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/4d20d8dcbd8a4f0e808fe0b86874bcff, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/e5a831d240cc4946ae03c60f2b9a9cf9] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp, totalSize=33.6 K 2023-03-29 03:54:55,062 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting f11c9143d9b541f5b7526fe38cb30d41, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=250, earliestPutTs=1680062093874 2023-03-29 03:54:55,062 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 5a2f8b2971fd4ceb99d780f2ff43116e, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=236, earliestPutTs=1680062093244 2023-03-29 03:54:55,062 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting e40baabca095450f9e91c9ee6137a521, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=265, earliestPutTs=1680062094997 2023-03-29 03:54:55,062 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 4d20d8dcbd8a4f0e808fe0b86874bcff, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=250, earliestPutTs=1680062093874 2023-03-29 03:54:55,063 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting e5a831d240cc4946ae03c60f2b9a9cf9, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=265, earliestPutTs=1680062094997 2023-03-29 03:54:55,070 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] throttle.PressureAwareThroughputController(145): 2ed55fa255954e1310f6b9fd96e28980#B#compaction#277 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:54:55,071 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2ed55fa255954e1310f6b9fd96e28980#A#compaction#278 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:54:55,086 DEBUG [Thread-1741] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:54:55,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:54:55,089 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:54:55,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:54:55,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:54:55,093 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:54:55,093 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:55,093 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:54:55,093 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:54:55,094 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:55,094 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:55,095 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:55,096 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:55,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:55,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:55,096 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:55,096 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:54:55,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:55,097 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:55,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:54:55,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:55,098 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:54:55,099 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:54:55,099 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:54:55,099 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:54:55,099 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:55,099 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:54:55,099 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:54:55,099 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:54:55,099 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:54:55,099 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. started... 2023-03-29 03:54:55,100 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2ed55fa255954e1310f6b9fd96e28980 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-29 03:54:55,100 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ed55fa255954e1310f6b9fd96e28980, store=A 2023-03-29 03:54:55,100 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:54:55,100 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ed55fa255954e1310f6b9fd96e28980, store=B 2023-03-29 03:54:55,100 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:54:55,100 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ed55fa255954e1310f6b9fd96e28980, store=C 2023-03-29 03:54:55,100 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/B/23c9506a31bb4fd28af54f0add74f389 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/23c9506a31bb4fd28af54f0add74f389 2023-03-29 03:54:55,100 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:54:55,109 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2ed55fa255954e1310f6b9fd96e28980/B of 2ed55fa255954e1310f6b9fd96e28980 into 23c9506a31bb4fd28af54f0add74f389(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:54:55,109 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2ed55fa255954e1310f6b9fd96e28980: 2023-03-29 03:54:55,109 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980., storeName=2ed55fa255954e1310f6b9fd96e28980/B, priority=13, startTime=1680062095059; duration=0sec 2023-03-29 03:54:55,109 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:54:55,109 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-29 03:54:55,110 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34360 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-29 03:54:55,110 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1912): 2ed55fa255954e1310f6b9fd96e28980/C is initiating minor compaction (all files) 2023-03-29 03:54:55,110 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2ed55fa255954e1310f6b9fd96e28980/C in TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:54:55,110 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/da7a8b072069472e9a74756697f88520, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/977142afe896430c8962ea3bdf5f054b, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/20059f61c4214d0891e5653b6d4dbd17] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp, totalSize=33.6 K 2023-03-29 03:54:55,111 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting da7a8b072069472e9a74756697f88520, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=236, earliestPutTs=1680062093244 2023-03-29 03:54:55,111 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 977142afe896430c8962ea3bdf5f054b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=250, earliestPutTs=1680062093874 2023-03-29 03:54:55,111 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 20059f61c4214d0891e5653b6d4dbd17, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=265, earliestPutTs=1680062094997 2023-03-29 03:54:55,119 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] throttle.PressureAwareThroughputController(145): 2ed55fa255954e1310f6b9fd96e28980#C#compaction#280 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:54:55,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on 2ed55fa255954e1310f6b9fd96e28980 2023-03-29 03:54:55,123 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. as already flushing 2023-03-29 03:54:55,132 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:55,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37610 deadline: 1680062155129, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:55,133 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:55,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37634 deadline: 1680062155130, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:55,133 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:55,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37570 deadline: 1680062155131, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:55,134 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:55,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37572 deadline: 1680062155132, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:55,145 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=285 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/A/7585ecb9f4564853b69199ca5babf598 2023-03-29 03:54:55,162 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=285 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/B/4be51dbc1b2f4086bc18a40fcb7aa564 2023-03-29 03:54:55,185 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=285 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/C/afce92aa5daa4ef3a32515a231e15986 2023-03-29 03:54:55,190 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/A/7585ecb9f4564853b69199ca5babf598 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/7585ecb9f4564853b69199ca5babf598 2023-03-29 03:54:55,194 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/7585ecb9f4564853b69199ca5babf598, entries=150, sequenceid=285, filesize=11.9 K 2023-03-29 03:54:55,195 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/B/4be51dbc1b2f4086bc18a40fcb7aa564 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/4be51dbc1b2f4086bc18a40fcb7aa564 2023-03-29 03:54:55,200 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/4be51dbc1b2f4086bc18a40fcb7aa564, entries=150, sequenceid=285, filesize=11.9 K 2023-03-29 03:54:55,200 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/C/afce92aa5daa4ef3a32515a231e15986 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/afce92aa5daa4ef3a32515a231e15986 2023-03-29 03:54:55,204 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/afce92aa5daa4ef3a32515a231e15986, entries=150, sequenceid=285, filesize=11.9 K 2023-03-29 03:54:55,205 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 2ed55fa255954e1310f6b9fd96e28980 in 105ms, sequenceid=285, compaction requested=false 2023-03-29 03:54:55,205 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2ed55fa255954e1310f6b9fd96e28980: 2023-03-29 03:54:55,205 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:54:55,205 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:54:55,205 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:54:55,205 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:55,205 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:54:55,205 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:54:55,208 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:55,208 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:55,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:55,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:55,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:55,208 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:55,208 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:54:55,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:55,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:55,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:55,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:55,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:55,210 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:54:55,210 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@78c78b96[Count = 0] remaining members to acquire global barrier 2023-03-29 03:54:55,210 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:54:55,210 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:55,211 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:55,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:55,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:55,211 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:54:55,211 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:55,211 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:54:55,211 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:54:55,211 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:54:55,213 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:54:55,213 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:55,213 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:54:55,213 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:55,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:55,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:55,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:55,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:55,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:55,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:55,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:55,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:55,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:55,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:55,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:54:55,216 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:54:55,216 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:54:55,216 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:54:55,216 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:54:55,216 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:54:55,216 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:54:55,218 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:55,218 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:55,218 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:55,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:55,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:55,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:55,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:55,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:55,219 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:55,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:55,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:55,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:55,219 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:55,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:55,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:55,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:55,221 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:55,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:55,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:55,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:55,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:55,225 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:55,225 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:55,225 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:55,225 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:55,225 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:54:55,225 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:54:55,225 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:54:55,225 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:55,225 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:55,225 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:55,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:55,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:55,226 DEBUG [Thread-1741] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:54:55,225 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:55,226 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:55,226 DEBUG [Thread-1741] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:54:55,226 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:55,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:55,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:55,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on 2ed55fa255954e1310f6b9fd96e28980 2023-03-29 03:54:55,234 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2ed55fa255954e1310f6b9fd96e28980 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-29 03:54:55,236 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ed55fa255954e1310f6b9fd96e28980, store=A 2023-03-29 03:54:55,236 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:54:55,236 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ed55fa255954e1310f6b9fd96e28980, store=B 2023-03-29 03:54:55,236 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:54:55,236 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ed55fa255954e1310f6b9fd96e28980, store=C 2023-03-29 03:54:55,236 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:54:55,244 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=300 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/A/cf5c7d1773db435ba216bb8382cf9b6a 2023-03-29 03:54:55,254 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:55,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37570 deadline: 1680062155251, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:55,256 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:55,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37610 deadline: 1680062155253, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:55,259 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:55,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37634 deadline: 1680062155255, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:55,259 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:55,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37572 deadline: 1680062155256, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:55,278 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=300 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/B/b31a4196edd24c88aed1c8646aafd41c 2023-03-29 03:54:55,326 DEBUG [Thread-1741] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:54:55,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:54:55,328 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:54:55,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:54:55,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:54:55,329 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:54:55,329 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:55,329 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:54:55,329 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:54:55,329 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:55,329 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:55,331 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:55,331 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:55,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:55,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:55,331 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:55,331 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:54:55,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:55,331 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:55,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:54:55,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:55,332 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:54:55,332 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:54:55,332 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:54:55,332 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:54:55,332 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:55,332 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:54:55,332 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:54:55,332 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:54:55,332 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:54:55,332 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. started... 2023-03-29 03:54:55,333 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. as already flushing 2023-03-29 03:54:55,333 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:54:55,333 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:54:55,333 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:54:55,333 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:55,333 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:54:55,333 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:54:55,334 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:55,334 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:55,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:55,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:55,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:55,334 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:55,335 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:54:55,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:55,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:55,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:55,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:55,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:55,336 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:54:55,336 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6bf9c9cc[Count = 0] remaining members to acquire global barrier 2023-03-29 03:54:55,336 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:54:55,336 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:55,337 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:55,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:55,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:55,337 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:54:55,337 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:54:55,337 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:54:55,337 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:55,337 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:54:55,340 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:54:55,340 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:55,340 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:55,340 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:54:55,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:55,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:55,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:55,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:55,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:55,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:55,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:55,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:55,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:55,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:55,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:54:55,342 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:54:55,342 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:54:55,342 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:54:55,342 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:54:55,342 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:54:55,342 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:54:55,344 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:55,344 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:55,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:55,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:55,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:55,344 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:55,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:55,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:55,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:55,344 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:55,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:55,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:55,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:55,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:55,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:55,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:55,345 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:55,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:55,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:55,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:55,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:55,347 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:55,347 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:55,347 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:54:55,347 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:54:55,347 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:55,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:55,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:55,347 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:55,348 DEBUG [Thread-1741] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:54:55,348 DEBUG [Thread-1741] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:54:55,347 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:55,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:55,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:55,348 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:55,348 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:54:55,348 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:55,348 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:55,348 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:55,348 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:55,356 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:55,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37570 deadline: 1680062155355, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:55,359 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:55,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37610 deadline: 1680062155357, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:55,363 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:55,363 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:55,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37634 deadline: 1680062155360, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:55,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37572 deadline: 1680062155360, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:55,448 DEBUG [Thread-1741] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:54:55,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:54:55,450 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:54:55,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:54:55,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:54:55,451 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:54:55,451 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:55,451 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:54:55,451 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:54:55,452 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:55,452 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:55,453 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:55,453 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:55,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:55,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:55,453 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:55,453 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:54:55,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:55,454 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:55,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:54:55,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:55,454 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:54:55,454 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:54:55,454 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:54:55,454 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:55,454 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:54:55,455 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:54:55,455 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:54:55,455 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:54:55,455 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. started... 2023-03-29 03:54:55,455 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. as already flushing 2023-03-29 03:54:55,455 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:54:55,455 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:54:55,455 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:54:55,455 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:55,455 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:54:55,455 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:54:55,458 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:55,458 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:55,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:55,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:55,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:55,458 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:55,458 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:54:55,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:55,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:55,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:55,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:55,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:55,459 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:54:55,459 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7d1b0f2a[Count = 0] remaining members to acquire global barrier 2023-03-29 03:54:55,459 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:54:55,459 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:55,460 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:55,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:55,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:55,460 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:54:55,460 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:55,460 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:54:55,460 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:54:55,461 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:54:55,462 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:54:55,462 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:55,462 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:55,462 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:54:55,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:55,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:55,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:55,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:55,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:55,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:55,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:55,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:55,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:55,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:55,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:54:55,464 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:54:55,464 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:54:55,464 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:54:55,464 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:54:55,464 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:54:55,464 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:54:55,465 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:55,465 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:55,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:55,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:55,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:55,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:55,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:55,465 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:55,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:55,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:55,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:55,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:55,466 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:55,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:55,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:55,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:55,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:55,467 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:55,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:55,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:55,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:55,470 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:55,470 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:55,470 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:55,470 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:55,470 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:55,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:55,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:55,470 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:54:55,471 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:54:55,470 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:55,471 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:55,471 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:55,471 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:55,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:55,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:55,471 DEBUG [Thread-1741] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:54:55,471 DEBUG [Thread-1741] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:54:55,471 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:55,490 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/A/96d79fb5dc124d8b93152ccf431fb98c as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/96d79fb5dc124d8b93152ccf431fb98c 2023-03-29 03:54:55,494 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2ed55fa255954e1310f6b9fd96e28980/A of 2ed55fa255954e1310f6b9fd96e28980 into 96d79fb5dc124d8b93152ccf431fb98c(size=12.5 K), total size for store is 24.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:54:55,494 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2ed55fa255954e1310f6b9fd96e28980: 2023-03-29 03:54:55,494 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980., storeName=2ed55fa255954e1310f6b9fd96e28980/A, priority=13, startTime=1680062095059; duration=0sec 2023-03-29 03:54:55,495 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:54:55,558 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:55,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37570 deadline: 1680062155557, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:55,562 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:55,562 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/C/a33da568269e4831816da56edc5f8cc6 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/a33da568269e4831816da56edc5f8cc6 2023-03-29 03:54:55,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37610 deadline: 1680062155560, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:55,571 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:55,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37634 deadline: 1680062155568, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:55,571 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2ed55fa255954e1310f6b9fd96e28980/C of 2ed55fa255954e1310f6b9fd96e28980 into a33da568269e4831816da56edc5f8cc6(size=12.5 K), total size for store is 24.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:54:55,571 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2ed55fa255954e1310f6b9fd96e28980: 2023-03-29 03:54:55,571 DEBUG [Thread-1741] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:54:55,571 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980., storeName=2ed55fa255954e1310f6b9fd96e28980/C, priority=13, startTime=1680062095059; duration=0sec 2023-03-29 03:54:55,571 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:55,572 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:54:55,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37572 deadline: 1680062155568, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:55,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:54:55,574 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:54:55,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:54:55,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:54:55,588 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:54:55,588 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:55,588 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:54:55,588 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:54:55,589 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:55,589 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:55,592 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:55,592 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:55,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:55,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:55,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:55,595 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:55,595 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:54:55,595 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:55,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:54:55,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:55,596 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:54:55,596 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:54:55,596 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:54:55,596 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:54:55,596 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:55,596 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:54:55,598 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:54:55,598 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:54:55,598 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:54:55,599 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. started... 2023-03-29 03:54:55,599 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. as already flushing 2023-03-29 03:54:55,599 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:54:55,599 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:54:55,599 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:54:55,599 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:55,599 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:54:55,599 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:54:55,600 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:55,600 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:55,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:55,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:55,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:55,601 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:55,601 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:54:55,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:55,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:55,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:55,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:55,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:55,603 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:54:55,603 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e80d8c2[Count = 0] remaining members to acquire global barrier 2023-03-29 03:54:55,603 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:54:55,603 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:55,606 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:55,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:55,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:55,606 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:54:55,606 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:54:55,606 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:54:55,606 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:55,606 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:54:55,607 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:55,607 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:54:55,607 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:55,607 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:54:55,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:55,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:55,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:55,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:55,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:55,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:55,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:55,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:55,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:55,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:55,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:54:55,610 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:54:55,610 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:54:55,610 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:54:55,610 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:54:55,610 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:54:55,610 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:54:55,612 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:55,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:55,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:55,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:55,612 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:55,612 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:55,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:55,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:55,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:55,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:55,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:55,613 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:55,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:55,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:55,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:55,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:55,614 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:55,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:55,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:55,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:55,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:55,616 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:55,616 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:55,616 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:55,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:55,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:55,616 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:55,616 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:54:55,616 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:55,616 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:54:55,616 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:54:55,616 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:55,616 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:55,616 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:55,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:55,617 DEBUG [Thread-1741] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:54:55,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:55,617 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:55,617 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:55,617 DEBUG [Thread-1741] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:54:55,654 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:55,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37614 deadline: 1680062155653, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:55,656 DEBUG [Thread-1739] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4155 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980., hostname=jenkins-hbase4.apache.org,35599,1680062018534, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-29 03:54:55,687 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=300 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/C/bcf82d1371cf4d788d18104ce5e497ba 2023-03-29 03:54:55,691 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/A/cf5c7d1773db435ba216bb8382cf9b6a as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/cf5c7d1773db435ba216bb8382cf9b6a 2023-03-29 03:54:55,695 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/cf5c7d1773db435ba216bb8382cf9b6a, entries=150, sequenceid=300, filesize=11.9 K 2023-03-29 03:54:55,695 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/B/b31a4196edd24c88aed1c8646aafd41c as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/b31a4196edd24c88aed1c8646aafd41c 2023-03-29 03:54:55,699 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/b31a4196edd24c88aed1c8646aafd41c, entries=100, sequenceid=300, filesize=9.5 K 2023-03-29 03:54:55,699 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/C/bcf82d1371cf4d788d18104ce5e497ba as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/bcf82d1371cf4d788d18104ce5e497ba 2023-03-29 03:54:55,702 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/bcf82d1371cf4d788d18104ce5e497ba, entries=100, sequenceid=300, filesize=9.5 K 2023-03-29 03:54:55,703 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=107.34 KB/109920 for 2ed55fa255954e1310f6b9fd96e28980 in 469ms, sequenceid=300, compaction requested=true 2023-03-29 03:54:55,703 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2ed55fa255954e1310f6b9fd96e28980: 2023-03-29 03:54:55,703 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:54:55,703 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:54:55,703 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-29 03:54:55,703 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:54:55,703 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-29 03:54:55,704 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37152 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-29 03:54:55,704 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1912): 2ed55fa255954e1310f6b9fd96e28980/A is initiating minor compaction (all files) 2023-03-29 03:54:55,704 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2ed55fa255954e1310f6b9fd96e28980/A in TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:54:55,704 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/96d79fb5dc124d8b93152ccf431fb98c, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/7585ecb9f4564853b69199ca5babf598, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/cf5c7d1773db435ba216bb8382cf9b6a] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp, totalSize=36.3 K 2023-03-29 03:54:55,705 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34712 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-29 03:54:55,705 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 96d79fb5dc124d8b93152ccf431fb98c, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=265, earliestPutTs=1680062093876 2023-03-29 03:54:55,705 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1912): 2ed55fa255954e1310f6b9fd96e28980/B is initiating minor compaction (all files) 2023-03-29 03:54:55,705 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2ed55fa255954e1310f6b9fd96e28980/B in TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:54:55,705 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/23c9506a31bb4fd28af54f0add74f389, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/4be51dbc1b2f4086bc18a40fcb7aa564, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/b31a4196edd24c88aed1c8646aafd41c] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp, totalSize=33.9 K 2023-03-29 03:54:55,705 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 7585ecb9f4564853b69199ca5babf598, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=285, earliestPutTs=1680062095015 2023-03-29 03:54:55,705 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 23c9506a31bb4fd28af54f0add74f389, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=265, earliestPutTs=1680062093876 2023-03-29 03:54:55,705 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting cf5c7d1773db435ba216bb8382cf9b6a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=300, earliestPutTs=1680062095130 2023-03-29 03:54:55,705 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 4be51dbc1b2f4086bc18a40fcb7aa564, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=285, earliestPutTs=1680062095015 2023-03-29 03:54:55,706 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting b31a4196edd24c88aed1c8646aafd41c, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=300, earliestPutTs=1680062095130 2023-03-29 03:54:55,711 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2ed55fa255954e1310f6b9fd96e28980#A#compaction#286 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:54:55,713 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] throttle.PressureAwareThroughputController(145): 2ed55fa255954e1310f6b9fd96e28980#B#compaction#287 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:54:55,717 DEBUG [Thread-1741] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:54:55,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:54:55,719 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:54:55,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:54:55,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:54:55,721 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:54:55,721 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:55,721 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:54:55,721 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:54:55,721 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:55,721 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:55,723 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/A/db0999310d304847bdb4d5cecf3f57f8 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/db0999310d304847bdb4d5cecf3f57f8 2023-03-29 03:54:55,724 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:55,724 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:55,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:55,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:55,724 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/B/e27e87bf31de4adfb06585b29ffbbcda as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/e27e87bf31de4adfb06585b29ffbbcda 2023-03-29 03:54:55,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:55,724 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:55,724 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:54:55,725 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:55,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:54:55,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:55,726 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:54:55,726 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:54:55,726 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:54:55,727 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:54:55,727 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:55,727 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:54:55,728 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:54:55,728 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:54:55,728 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:54:55,728 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. started... 2023-03-29 03:54:55,728 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2ed55fa255954e1310f6b9fd96e28980 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-29 03:54:55,728 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ed55fa255954e1310f6b9fd96e28980, store=A 2023-03-29 03:54:55,728 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:54:55,728 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ed55fa255954e1310f6b9fd96e28980, store=B 2023-03-29 03:54:55,728 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:54:55,729 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ed55fa255954e1310f6b9fd96e28980, store=C 2023-03-29 03:54:55,729 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:54:55,743 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2ed55fa255954e1310f6b9fd96e28980/A of 2ed55fa255954e1310f6b9fd96e28980 into db0999310d304847bdb4d5cecf3f57f8(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:54:55,743 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2ed55fa255954e1310f6b9fd96e28980: 2023-03-29 03:54:55,743 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980., storeName=2ed55fa255954e1310f6b9fd96e28980/A, priority=13, startTime=1680062095703; duration=0sec 2023-03-29 03:54:55,744 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:54:55,744 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2ed55fa255954e1310f6b9fd96e28980/B of 2ed55fa255954e1310f6b9fd96e28980 into e27e87bf31de4adfb06585b29ffbbcda(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:54:55,744 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-29 03:54:55,744 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2ed55fa255954e1310f6b9fd96e28980: 2023-03-29 03:54:55,744 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980., storeName=2ed55fa255954e1310f6b9fd96e28980/B, priority=13, startTime=1680062095703; duration=0sec 2023-03-29 03:54:55,744 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:54:55,745 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34712 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-29 03:54:55,745 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1912): 2ed55fa255954e1310f6b9fd96e28980/C is initiating minor compaction (all files) 2023-03-29 03:54:55,745 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2ed55fa255954e1310f6b9fd96e28980/C in TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:54:55,745 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/a33da568269e4831816da56edc5f8cc6, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/afce92aa5daa4ef3a32515a231e15986, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/bcf82d1371cf4d788d18104ce5e497ba] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp, totalSize=33.9 K 2023-03-29 03:54:55,745 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting a33da568269e4831816da56edc5f8cc6, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=265, earliestPutTs=1680062093876 2023-03-29 03:54:55,746 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting afce92aa5daa4ef3a32515a231e15986, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=285, earliestPutTs=1680062095015 2023-03-29 03:54:55,746 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting bcf82d1371cf4d788d18104ce5e497ba, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=300, earliestPutTs=1680062095130 2023-03-29 03:54:55,751 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2ed55fa255954e1310f6b9fd96e28980#C#compaction#289 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:54:55,759 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/C/ff30a175e2014f7287d14a2e90f29d98 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/ff30a175e2014f7287d14a2e90f29d98 2023-03-29 03:54:55,763 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2ed55fa255954e1310f6b9fd96e28980/C of 2ed55fa255954e1310f6b9fd96e28980 into ff30a175e2014f7287d14a2e90f29d98(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:54:55,763 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2ed55fa255954e1310f6b9fd96e28980: 2023-03-29 03:54:55,763 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980., storeName=2ed55fa255954e1310f6b9fd96e28980/C, priority=13, startTime=1680062095703; duration=0sec 2023-03-29 03:54:55,763 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:54:55,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on 2ed55fa255954e1310f6b9fd96e28980 2023-03-29 03:54:55,861 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. as already flushing 2023-03-29 03:54:55,896 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:55,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37572 deadline: 1680062155875, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:55,898 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:55,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37570 deadline: 1680062155896, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:55,898 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:55,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37610 deadline: 1680062155896, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:55,898 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:55,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37634 deadline: 1680062155896, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:55,999 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:55,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37610 deadline: 1680062155999, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:55,999 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:55,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37570 deadline: 1680062155999, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:56,003 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:56,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37634 deadline: 1680062156001, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:56,151 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=321 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/A/607b40de47d04783a23e000d67520b6f 2023-03-29 03:54:56,161 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=321 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/B/c209d746f09f47df935bae6ac4905037 2023-03-29 03:54:56,201 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:56,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37570 deadline: 1680062156200, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:56,202 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:56,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37610 deadline: 1680062156201, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:56,206 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:56,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37634 deadline: 1680062156205, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:56,402 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:56,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37572 deadline: 1680062156401, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:56,505 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:56,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37570 deadline: 1680062156503, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:56,506 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:56,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37610 deadline: 1680062156505, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:56,509 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:56,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37634 deadline: 1680062156508, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:56,570 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=321 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/C/0a840e43c8fa48c9b41d949254f71fc5 2023-03-29 03:54:56,576 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/A/607b40de47d04783a23e000d67520b6f as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/607b40de47d04783a23e000d67520b6f 2023-03-29 03:54:56,579 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/607b40de47d04783a23e000d67520b6f, entries=150, sequenceid=321, filesize=11.9 K 2023-03-29 03:54:56,580 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/B/c209d746f09f47df935bae6ac4905037 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/c209d746f09f47df935bae6ac4905037 2023-03-29 03:54:56,584 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/c209d746f09f47df935bae6ac4905037, entries=150, sequenceid=321, filesize=11.9 K 2023-03-29 03:54:56,585 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/C/0a840e43c8fa48c9b41d949254f71fc5 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/0a840e43c8fa48c9b41d949254f71fc5 2023-03-29 03:54:56,589 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/0a840e43c8fa48c9b41d949254f71fc5, entries=150, sequenceid=321, filesize=11.9 K 2023-03-29 03:54:56,590 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=80.51 KB/82440 for 2ed55fa255954e1310f6b9fd96e28980 in 862ms, sequenceid=321, compaction requested=false 2023-03-29 03:54:56,590 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2ed55fa255954e1310f6b9fd96e28980: 2023-03-29 03:54:56,590 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:54:56,590 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:54:56,590 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:54:56,590 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:56,590 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:54:56,590 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:54:56,592 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:56,592 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:56,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:56,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:56,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:56,592 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:56,592 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:54:56,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:56,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:56,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:56,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:56,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:56,594 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:54:56,594 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7adf7fbf[Count = 0] remaining members to acquire global barrier 2023-03-29 03:54:56,594 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:54:56,594 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:56,595 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:56,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:56,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:56,595 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:54:56,595 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:54:56,595 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:54:56,595 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:56,595 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:54:56,597 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:56,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:56,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:56,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:56,597 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:54:56,597 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:56,597 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:54:56,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:56,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:56,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:56,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:56,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:56,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:56,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:56,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:54:56,599 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:54:56,599 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:54:56,599 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:54:56,599 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:54:56,599 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:54:56,599 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:54:56,602 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:56,602 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:56,602 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:56,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:56,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:56,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:56,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:56,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:56,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:56,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:56,603 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:56,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:56,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:56,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:56,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:56,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:56,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:56,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:56,604 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:56,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:56,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:56,607 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:56,607 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:56,607 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:56,607 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:54:56,607 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:54:56,607 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:56,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:56,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:56,607 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:56,607 DEBUG [Thread-1741] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:54:56,607 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:56,607 DEBUG [Thread-1741] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:54:56,607 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:56,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:56,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:56,608 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:56,608 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:56,608 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:56,708 DEBUG [Thread-1741] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:54:56,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:54:56,710 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:54:56,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:54:56,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:54:56,711 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:54:56,711 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:56,711 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:54:56,711 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:54:56,711 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:56,711 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:56,712 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:56,712 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:56,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:56,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:56,713 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:56,713 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:54:56,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:56,713 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:56,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:54:56,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:56,713 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:54:56,714 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:54:56,714 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:54:56,714 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:54:56,714 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:56,714 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:54:56,714 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:54:56,714 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:54:56,714 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:54:56,714 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. started... 2023-03-29 03:54:56,714 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2ed55fa255954e1310f6b9fd96e28980 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-03-29 03:54:56,714 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ed55fa255954e1310f6b9fd96e28980, store=A 2023-03-29 03:54:56,714 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:54:56,714 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ed55fa255954e1310f6b9fd96e28980, store=B 2023-03-29 03:54:56,714 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:54:56,714 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ed55fa255954e1310f6b9fd96e28980, store=C 2023-03-29 03:54:56,715 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:54:56,722 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=339 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/A/399637a46e644dceb34922f1d2962c6f 2023-03-29 03:54:56,731 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=339 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/B/0ffd1d8e06b940e2b5bba5f3a72e2958 2023-03-29 03:54:56,754 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=339 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/C/75e5c22bbee4423a9a0c04eade018515 2023-03-29 03:54:56,758 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/A/399637a46e644dceb34922f1d2962c6f as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/399637a46e644dceb34922f1d2962c6f 2023-03-29 03:54:56,762 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/399637a46e644dceb34922f1d2962c6f, entries=150, sequenceid=339, filesize=11.9 K 2023-03-29 03:54:56,762 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/B/0ffd1d8e06b940e2b5bba5f3a72e2958 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/0ffd1d8e06b940e2b5bba5f3a72e2958 2023-03-29 03:54:56,766 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/0ffd1d8e06b940e2b5bba5f3a72e2958, entries=150, sequenceid=339, filesize=11.9 K 2023-03-29 03:54:56,766 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/C/75e5c22bbee4423a9a0c04eade018515 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/75e5c22bbee4423a9a0c04eade018515 2023-03-29 03:54:56,769 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/75e5c22bbee4423a9a0c04eade018515, entries=150, sequenceid=339, filesize=11.9 K 2023-03-29 03:54:56,770 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=0 B/0 for 2ed55fa255954e1310f6b9fd96e28980 in 56ms, sequenceid=339, compaction requested=true 2023-03-29 03:54:56,770 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2ed55fa255954e1310f6b9fd96e28980: 2023-03-29 03:54:56,770 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:54:56,770 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:54:56,770 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:54:56,770 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:56,770 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:54:56,770 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:54:56,772 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:56,772 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:56,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:56,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:56,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:56,772 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:56,772 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:54:56,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:56,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:56,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:56,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:56,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:56,774 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:54:56,774 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5fcd7e81[Count = 0] remaining members to acquire global barrier 2023-03-29 03:54:56,774 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:54:56,774 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:56,776 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:56,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:56,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:56,776 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:54:56,777 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:54:56,777 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:54:56,777 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:56,777 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:54:56,778 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:56,778 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:54:56,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:56,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:56,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:56,778 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:56,778 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:54:56,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:56,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:56,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:56,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:56,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:56,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:56,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:56,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:54:56,780 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:54:56,780 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:54:56,781 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:54:56,781 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:54:56,781 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:54:56,781 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:54:56,782 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:56,782 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:56,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:56,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:56,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:56,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:56,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:56,782 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:56,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:56,783 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:56,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:56,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:56,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:56,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:56,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:56,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:56,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:56,784 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:56,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:56,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:56,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:56,788 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:56,788 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:56,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:56,788 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:56,788 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:56,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:56,788 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:54:56,788 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:54:56,788 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:56,788 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:56,788 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:56,788 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:56,788 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:56,788 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:56,788 DEBUG [Thread-1741] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:54:56,788 DEBUG [Thread-1741] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:54:56,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:56,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:56,788 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:54:56,888 DEBUG [Thread-1741] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:54:56,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:54:56,890 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:54:56,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:54:56,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:54:56,892 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:54:56,892 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:56,892 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:54:56,892 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:54:56,893 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:56,893 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:56,904 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:56,905 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:56,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:56,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:56,905 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:56,905 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:54:56,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:56,905 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:56,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:54:56,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:56,906 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:54:56,906 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:54:56,906 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:54:56,906 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:54:56,906 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:56,906 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:54:56,906 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:54:56,906 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:54:56,906 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:54:56,907 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. started... 2023-03-29 03:54:56,907 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2ed55fa255954e1310f6b9fd96e28980: 2023-03-29 03:54:56,907 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:54:56,907 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:54:56,907 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:54:56,907 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:56,907 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:54:56,907 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:54:56,910 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:56,910 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:56,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:56,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:56,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:56,911 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:56,911 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:54:56,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:56,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:56,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:56,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:56,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:56,912 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:54:56,912 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a8a480d[Count = 0] remaining members to acquire global barrier 2023-03-29 03:54:56,912 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:54:56,912 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:56,913 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:56,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:56,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:56,914 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:54:56,914 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:54:56,914 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:54:56,914 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:56,914 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:54:56,916 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:54:56,916 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:56,916 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:54:56,916 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:56,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:56,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:56,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:56,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:56,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:56,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:56,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:56,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:56,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:56,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:56,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:54:56,919 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:54:56,919 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:54:56,919 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:54:56,919 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:54:56,919 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:54:56,919 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:54:56,920 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:56,920 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:56,920 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:56,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:56,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:56,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:56,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:56,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:56,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:56,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:56,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:56,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:56,921 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:56,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:56,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:56,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:56,922 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:56,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:56,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:56,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:56,925 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:56,925 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:56,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:56,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:56,925 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:56,925 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:56,925 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:54:56,925 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:54:56,925 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:56,925 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:56,925 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:56,925 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:56,925 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:56,925 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:56,925 DEBUG [Thread-1741] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:54:56,925 DEBUG [Thread-1741] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:54:56,926 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:54:56,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:56,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:57,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on 2ed55fa255954e1310f6b9fd96e28980 2023-03-29 03:54:57,017 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2ed55fa255954e1310f6b9fd96e28980 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-29 03:54:57,017 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ed55fa255954e1310f6b9fd96e28980, store=A 2023-03-29 03:54:57,017 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:54:57,017 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ed55fa255954e1310f6b9fd96e28980, store=B 2023-03-29 03:54:57,017 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:54:57,017 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ed55fa255954e1310f6b9fd96e28980, store=C 2023-03-29 03:54:57,017 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:54:57,026 DEBUG [Thread-1741] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:54:57,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:54:57,029 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:54:57,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:54:57,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:54:57,031 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:54:57,031 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:57,031 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:54:57,032 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:54:57,032 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:57,032 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:57,033 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:57,033 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:57,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:57,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:57,034 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:57,034 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:54:57,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:57,034 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:57,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:54:57,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:57,035 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:54:57,036 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:54:57,036 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:54:57,036 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:54:57,036 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:57,036 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:54:57,036 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:54:57,036 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:54:57,036 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:54:57,036 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. started... 2023-03-29 03:54:57,036 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. as already flushing 2023-03-29 03:54:57,036 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:54:57,036 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:54:57,036 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:54:57,036 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:57,036 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:54:57,037 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:54:57,039 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:57,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:57,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:57,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:57,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:57,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:57,040 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:57,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:57,040 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:57,040 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:54:57,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:57,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:57,042 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:54:57,042 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@42f4c956[Count = 0] remaining members to acquire global barrier 2023-03-29 03:54:57,042 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:54:57,042 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:57,043 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:57,043 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:57,043 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:54:57,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:57,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:57,044 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:54:57,044 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:54:57,044 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:54:57,045 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:57,045 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:54:57,045 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:57,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:57,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:57,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:57,045 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:54:57,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:57,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:57,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:57,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:57,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:57,047 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680062097037, max timestamp=1680062097046Num uniques 859; 2023-03-29 03:54:57,048 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680062097037, max timestamp=1680062097046Num uniques 859; 2023-03-29 03:54:57,048 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680062097037, max timestamp=1680062097046Num uniques 859; 2023-03-29 03:54:57,048 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-29 03:54:57,048 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-29 03:54:57,048 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-29 03:54:57,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:57,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:57,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:54:57,049 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:54:57,049 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:54:57,049 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:54:57,049 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:54:57,049 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:54:57,049 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:54:57,050 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:57,051 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:57,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:57,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:57,051 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:57,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:57,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:57,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:57,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:57,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:57,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:57,051 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:57,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:57,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:57,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:57,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:57,053 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:57,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:57,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:57,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:57,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:57,057 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:57,057 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:57,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:57,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:57,057 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:57,057 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:54:57,057 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:54:57,057 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:54:57,057 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:57,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:57,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:57,057 DEBUG [Thread-1741] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:54:57,057 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:57,057 DEBUG [Thread-1741] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:54:57,057 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:57,057 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:57,057 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:57,057 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:57,058 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:57,071 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:57,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37634 deadline: 1680062157067, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:57,071 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:57,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37570 deadline: 1680062157069, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:57,073 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:57,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37610 deadline: 1680062157070, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:57,157 DEBUG [Thread-1741] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:54:57,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:54:57,159 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:54:57,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:54:57,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:54:57,161 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:54:57,161 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:57,161 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:54:57,161 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:54:57,161 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:57,161 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:57,162 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:57,162 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:57,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:57,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:57,163 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:57,163 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:54:57,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:57,163 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:57,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:54:57,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:57,164 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:54:57,164 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:54:57,164 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:54:57,164 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:54:57,164 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:57,164 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:54:57,164 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:54:57,164 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:54:57,164 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:54:57,164 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. started... 2023-03-29 03:54:57,164 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. as already flushing 2023-03-29 03:54:57,164 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:54:57,164 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:54:57,165 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:54:57,165 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:57,165 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:54:57,165 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:54:57,166 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:57,166 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:57,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:57,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:57,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:57,166 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:57,166 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:54:57,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:57,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:57,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:57,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:57,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:57,168 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:54:57,168 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@43d76430[Count = 0] remaining members to acquire global barrier 2023-03-29 03:54:57,168 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:54:57,168 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:57,169 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:57,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:57,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:57,169 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:54:57,169 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:54:57,169 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:54:57,169 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:57,169 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:54:57,172 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:57,172 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:54:57,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:57,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:57,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:57,172 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:57,172 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:54:57,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:57,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:57,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:57,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:57,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:57,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:57,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:57,174 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:57,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37634 deadline: 1680062157172, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:57,174 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:57,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:54:57,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37570 deadline: 1680062157172, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:57,175 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:54:57,175 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:54:57,175 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:54:57,175 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:54:57,175 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:54:57,175 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:54:57,175 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:57,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37610 deadline: 1680062157174, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:57,177 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:57,177 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:57,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:57,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:57,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:57,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:57,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:57,177 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:57,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:57,177 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:57,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:57,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:57,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:57,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:57,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:57,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:57,179 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:57,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:57,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:57,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:57,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:57,181 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:57,181 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:54:57,181 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:54:57,181 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:54:57,181 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:57,181 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:57,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:57,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:57,181 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:57,181 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:57,181 DEBUG [Thread-1741] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:54:57,181 DEBUG [Thread-1741] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:54:57,181 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:57,182 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:57,182 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:57,182 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:57,182 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:57,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:57,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:57,282 DEBUG [Thread-1741] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:54:57,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:54:57,284 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:54:57,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:54:57,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:54:57,285 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:54:57,285 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:57,285 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:54:57,285 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:54:57,286 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:57,286 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:57,287 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:57,287 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:57,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:57,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:57,287 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:57,287 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:54:57,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:57,288 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:57,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:54:57,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:57,288 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:54:57,289 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:54:57,289 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:54:57,289 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:54:57,289 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:57,289 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:54:57,289 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:54:57,289 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:54:57,289 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:54:57,289 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. started... 2023-03-29 03:54:57,289 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. as already flushing 2023-03-29 03:54:57,289 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:54:57,289 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:54:57,289 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:54:57,289 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:57,289 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:54:57,289 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:54:57,292 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:57,292 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:57,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:57,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:57,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:57,293 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:57,293 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:54:57,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:57,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:57,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:57,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:57,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:57,294 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:54:57,294 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@77da23bd[Count = 0] remaining members to acquire global barrier 2023-03-29 03:54:57,294 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:54:57,294 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:57,295 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:57,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:57,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:57,295 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:54:57,295 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:54:57,295 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:54:57,296 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:57,296 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:54:57,299 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:57,299 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:54:57,299 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:57,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:57,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:57,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:57,299 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:54:57,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:57,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:57,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:57,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:57,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:57,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:57,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:57,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:54:57,302 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:54:57,302 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:54:57,302 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:54:57,302 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:54:57,302 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:54:57,302 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:54:57,304 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:57,304 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:57,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:57,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:57,304 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:57,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:57,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:57,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:57,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:57,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:57,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:57,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:57,305 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:57,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:57,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:57,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:57,306 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:57,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:57,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:57,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:57,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:57,308 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:57,308 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:57,308 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:57,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:57,308 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:57,308 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:54:57,309 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:54:57,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:57,308 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:57,309 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:57,309 DEBUG [Thread-1741] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:54:57,309 DEBUG [Thread-1741] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:54:57,309 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:57,309 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:57,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:57,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:57,309 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:57,309 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:57,376 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:57,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37634 deadline: 1680062157375, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:57,377 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:57,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37610 deadline: 1680062157376, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:57,377 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:57,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37570 deadline: 1680062157377, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:57,406 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:57,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37572 deadline: 1680062157406, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:57,409 DEBUG [Thread-1741] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:54:57,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:54:57,411 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:54:57,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:54:57,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:54:57,412 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:54:57,412 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:57,412 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:54:57,412 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:54:57,413 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:57,413 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:57,414 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:57,414 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:57,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:57,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:57,414 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:57,414 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:54:57,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:57,415 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:57,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:54:57,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:57,415 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:54:57,415 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:54:57,415 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:54:57,415 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:54:57,415 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:57,415 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:54:57,416 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:54:57,416 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:54:57,416 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:54:57,416 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. started... 2023-03-29 03:54:57,416 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. as already flushing 2023-03-29 03:54:57,416 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:54:57,416 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:54:57,416 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:54:57,416 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:57,416 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:54:57,416 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:54:57,417 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:57,417 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:57,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:57,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:57,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:57,418 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:57,418 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:54:57,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:57,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:57,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:57,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:57,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:57,419 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:54:57,419 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:54:57,419 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f38c76b[Count = 0] remaining members to acquire global barrier 2023-03-29 03:54:57,419 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:57,421 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:57,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:57,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:57,421 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:54:57,421 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:54:57,421 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:57,421 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:54:57,421 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:54:57,423 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:57,423 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:54:57,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:57,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:57,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:57,423 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:57,423 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:54:57,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:57,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:57,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:57,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:57,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:57,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:57,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:57,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:54:57,425 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:54:57,425 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:54:57,425 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:54:57,425 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:54:57,425 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:54:57,425 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:54:57,427 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:57,427 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:57,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:57,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:57,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:57,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:57,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:57,427 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:57,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:57,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:57,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:57,428 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:57,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:57,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:57,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:57,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:57,429 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:57,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:57,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:57,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:57,431 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=351 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/A/1f64b87f85074eb5b6abbcbbae1b86eb 2023-03-29 03:54:57,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:57,432 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:57,432 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:57,432 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:57,432 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:57,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:57,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:57,432 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:57,432 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:57,433 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:57,433 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:57,433 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:57,433 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:57,433 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:54:57,433 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:54:57,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:57,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:57,433 DEBUG [Thread-1741] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:54:57,433 DEBUG [Thread-1741] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:54:57,436 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:54:57,446 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=351 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/B/7fd9d678aa4a4150afd006f684754c24 2023-03-29 03:54:57,471 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=351 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/C/eb29b1a2903b4523baab76616d7b596c 2023-03-29 03:54:57,477 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/A/1f64b87f85074eb5b6abbcbbae1b86eb as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/1f64b87f85074eb5b6abbcbbae1b86eb 2023-03-29 03:54:57,482 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/1f64b87f85074eb5b6abbcbbae1b86eb, entries=250, sequenceid=351, filesize=16.7 K 2023-03-29 03:54:57,483 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/B/7fd9d678aa4a4150afd006f684754c24 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/7fd9d678aa4a4150afd006f684754c24 2023-03-29 03:54:57,487 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/7fd9d678aa4a4150afd006f684754c24, entries=150, sequenceid=351, filesize=11.9 K 2023-03-29 03:54:57,488 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/C/eb29b1a2903b4523baab76616d7b596c as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/eb29b1a2903b4523baab76616d7b596c 2023-03-29 03:54:57,492 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/eb29b1a2903b4523baab76616d7b596c, entries=150, sequenceid=351, filesize=11.9 K 2023-03-29 03:54:57,493 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 2ed55fa255954e1310f6b9fd96e28980 in 476ms, sequenceid=351, compaction requested=true 2023-03-29 03:54:57,493 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2ed55fa255954e1310f6b9fd96e28980: 2023-03-29 03:54:57,493 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:54:57,493 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:54:57,493 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-29 03:54:57,493 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-29 03:54:57,493 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:54:57,494 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 54374 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-29 03:54:57,495 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49494 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-29 03:54:57,495 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1912): 2ed55fa255954e1310f6b9fd96e28980/A is initiating minor compaction (all files) 2023-03-29 03:54:57,495 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1912): 2ed55fa255954e1310f6b9fd96e28980/B is initiating minor compaction (all files) 2023-03-29 03:54:57,495 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2ed55fa255954e1310f6b9fd96e28980/A in TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:54:57,495 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2ed55fa255954e1310f6b9fd96e28980/B in TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:54:57,495 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/db0999310d304847bdb4d5cecf3f57f8, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/607b40de47d04783a23e000d67520b6f, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/399637a46e644dceb34922f1d2962c6f, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/1f64b87f85074eb5b6abbcbbae1b86eb] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp, totalSize=53.1 K 2023-03-29 03:54:57,495 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/e27e87bf31de4adfb06585b29ffbbcda, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/c209d746f09f47df935bae6ac4905037, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/0ffd1d8e06b940e2b5bba5f3a72e2958, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/7fd9d678aa4a4150afd006f684754c24] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp, totalSize=48.3 K 2023-03-29 03:54:57,495 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting db0999310d304847bdb4d5cecf3f57f8, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=300, earliestPutTs=1680062095015 2023-03-29 03:54:57,495 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting e27e87bf31de4adfb06585b29ffbbcda, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=300, earliestPutTs=1680062095015 2023-03-29 03:54:57,495 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 607b40de47d04783a23e000d67520b6f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=321, earliestPutTs=1680062095252 2023-03-29 03:54:57,496 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting c209d746f09f47df935bae6ac4905037, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=321, earliestPutTs=1680062095252 2023-03-29 03:54:57,496 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 399637a46e644dceb34922f1d2962c6f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=339, earliestPutTs=1680062095868 2023-03-29 03:54:57,496 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 0ffd1d8e06b940e2b5bba5f3a72e2958, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=339, earliestPutTs=1680062095868 2023-03-29 03:54:57,496 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 1f64b87f85074eb5b6abbcbbae1b86eb, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=351, earliestPutTs=1680062097012 2023-03-29 03:54:57,496 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 7fd9d678aa4a4150afd006f684754c24, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=351, earliestPutTs=1680062097012 2023-03-29 03:54:57,504 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2ed55fa255954e1310f6b9fd96e28980#A#compaction#298 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:54:57,504 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] throttle.PressureAwareThroughputController(145): 2ed55fa255954e1310f6b9fd96e28980#B#compaction#299 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:54:57,533 DEBUG [Thread-1741] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:54:57,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:54:57,535 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:54:57,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:54:57,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:54:57,536 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:54:57,536 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:57,536 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:54:57,536 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:54:57,537 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:57,537 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:57,538 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:57,538 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:57,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:57,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:57,539 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:57,539 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:54:57,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:57,539 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:57,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:54:57,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:57,539 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:54:57,540 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:54:57,540 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:54:57,540 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:54:57,540 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:57,540 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:54:57,540 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:54:57,540 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:54:57,540 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:54:57,540 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. started... 2023-03-29 03:54:57,540 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2ed55fa255954e1310f6b9fd96e28980 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-03-29 03:54:57,540 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ed55fa255954e1310f6b9fd96e28980, store=A 2023-03-29 03:54:57,540 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-29 03:54:57,541 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ed55fa255954e1310f6b9fd96e28980, store=B 2023-03-29 03:54:57,541 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-29 03:54:57,541 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ed55fa255954e1310f6b9fd96e28980, store=C 2023-03-29 03:54:57,541 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-29 03:54:57,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on 2ed55fa255954e1310f6b9fd96e28980 2023-03-29 03:54:57,681 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. as already flushing 2023-03-29 03:54:57,694 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:57,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37570 deadline: 1680062157689, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:57,694 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:57,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37610 deadline: 1680062157690, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:57,696 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:57,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37634 deadline: 1680062157692, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:57,796 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:57,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37570 deadline: 1680062157795, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:57,798 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:57,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37610 deadline: 1680062157795, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:57,798 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:57,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37634 deadline: 1680062157797, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:57,914 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/A/98e78e88fa334457a10d8d8bafcc31ac as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/98e78e88fa334457a10d8d8bafcc31ac 2023-03-29 03:54:57,919 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 2ed55fa255954e1310f6b9fd96e28980/A of 2ed55fa255954e1310f6b9fd96e28980 into 98e78e88fa334457a10d8d8bafcc31ac(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:54:57,919 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2ed55fa255954e1310f6b9fd96e28980: 2023-03-29 03:54:57,919 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980., storeName=2ed55fa255954e1310f6b9fd96e28980/A, priority=12, startTime=1680062097493; duration=0sec 2023-03-29 03:54:57,919 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/B/1b6c5a6406044890b8993cc36b060672 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/1b6c5a6406044890b8993cc36b060672 2023-03-29 03:54:57,920 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:54:57,920 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-29 03:54:57,921 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49494 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-29 03:54:57,922 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1912): 2ed55fa255954e1310f6b9fd96e28980/C is initiating minor compaction (all files) 2023-03-29 03:54:57,922 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2ed55fa255954e1310f6b9fd96e28980/C in TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:54:57,922 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/ff30a175e2014f7287d14a2e90f29d98, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/0a840e43c8fa48c9b41d949254f71fc5, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/75e5c22bbee4423a9a0c04eade018515, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/eb29b1a2903b4523baab76616d7b596c] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp, totalSize=48.3 K 2023-03-29 03:54:57,922 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting ff30a175e2014f7287d14a2e90f29d98, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=300, earliestPutTs=1680062095015 2023-03-29 03:54:57,923 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 0a840e43c8fa48c9b41d949254f71fc5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=321, earliestPutTs=1680062095252 2023-03-29 03:54:57,923 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 75e5c22bbee4423a9a0c04eade018515, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=339, earliestPutTs=1680062095868 2023-03-29 03:54:57,924 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting eb29b1a2903b4523baab76616d7b596c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=351, earliestPutTs=1680062097012 2023-03-29 03:54:57,925 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 2ed55fa255954e1310f6b9fd96e28980/B of 2ed55fa255954e1310f6b9fd96e28980 into 1b6c5a6406044890b8993cc36b060672(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:54:57,925 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2ed55fa255954e1310f6b9fd96e28980: 2023-03-29 03:54:57,925 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980., storeName=2ed55fa255954e1310f6b9fd96e28980/B, priority=12, startTime=1680062097493; duration=0sec 2023-03-29 03:54:57,925 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:54:57,946 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2ed55fa255954e1310f6b9fd96e28980#C#compaction#301 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:54:57,947 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=386 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/A/932e0c76313543b29e70d1b8be69fff8 2023-03-29 03:54:57,975 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=386 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/B/b23952d5900849b693e69ff429be80c3 2023-03-29 03:54:57,977 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/C/78dd16f5f1c441d08f5100f41a11211e as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/78dd16f5f1c441d08f5100f41a11211e 2023-03-29 03:54:57,982 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 2ed55fa255954e1310f6b9fd96e28980/C of 2ed55fa255954e1310f6b9fd96e28980 into 78dd16f5f1c441d08f5100f41a11211e(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:54:57,982 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2ed55fa255954e1310f6b9fd96e28980: 2023-03-29 03:54:57,982 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980., storeName=2ed55fa255954e1310f6b9fd96e28980/C, priority=12, startTime=1680062097493; duration=0sec 2023-03-29 03:54:57,983 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:54:57,999 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:57,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37570 deadline: 1680062157998, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:58,000 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:58,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37610 deadline: 1680062157999, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:58,001 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:58,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37634 deadline: 1680062158000, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:58,301 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:58,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37570 deadline: 1680062158301, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:58,304 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:58,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37610 deadline: 1680062158303, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:58,305 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:58,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37634 deadline: 1680062158304, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:58,402 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=386 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/C/7f5458004d224d53b183f0cca1bdf866 2023-03-29 03:54:58,406 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/A/932e0c76313543b29e70d1b8be69fff8 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/932e0c76313543b29e70d1b8be69fff8 2023-03-29 03:54:58,409 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/932e0c76313543b29e70d1b8be69fff8, entries=150, sequenceid=386, filesize=11.9 K 2023-03-29 03:54:58,409 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.StoreScanner(961): StoreScanner already closing. There is no need to updateReaders 2023-03-29 03:54:58,410 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/B/b23952d5900849b693e69ff429be80c3 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/b23952d5900849b693e69ff429be80c3 2023-03-29 03:54:58,414 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/b23952d5900849b693e69ff429be80c3, entries=150, sequenceid=386, filesize=11.9 K 2023-03-29 03:54:58,415 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/C/7f5458004d224d53b183f0cca1bdf866 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/7f5458004d224d53b183f0cca1bdf866 2023-03-29 03:54:58,418 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/7f5458004d224d53b183f0cca1bdf866, entries=150, sequenceid=386, filesize=11.9 K 2023-03-29 03:54:58,419 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=53.67 KB/54960 for 2ed55fa255954e1310f6b9fd96e28980 in 879ms, sequenceid=386, compaction requested=false 2023-03-29 03:54:58,419 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2ed55fa255954e1310f6b9fd96e28980: 2023-03-29 03:54:58,419 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:54:58,419 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:54:58,419 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:54:58,419 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:58,419 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:54:58,419 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:54:58,421 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:58,421 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:58,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:58,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:58,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:58,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:58,421 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:58,421 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:54:58,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:58,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:58,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:58,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:58,423 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:54:58,423 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3b620ba3[Count = 0] remaining members to acquire global barrier 2023-03-29 03:54:58,423 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:54:58,423 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:58,424 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:58,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:58,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:58,424 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:54:58,424 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:54:58,424 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:54:58,424 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:58,424 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:54:58,426 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:54:58,426 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:58,426 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:58,426 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:54:58,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:58,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:58,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:58,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:58,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:58,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:58,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:58,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:58,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:58,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:58,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:54:58,428 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:54:58,428 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:54:58,428 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:54:58,428 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:54:58,428 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:54:58,428 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:54:58,431 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:58,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:58,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:58,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:58,431 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:58,431 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:58,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:58,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:58,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:58,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:58,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:58,432 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:58,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:58,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:58,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:58,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:58,433 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:58,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:58,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:58,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:58,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:58,435 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:58,435 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:58,435 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:58,435 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:54:58,435 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:54:58,435 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:58,435 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:58,435 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:54:58,435 DEBUG [Thread-1741] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:54:58,435 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:58,436 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:58,436 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:58,436 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:58,436 DEBUG [Thread-1741] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:54:58,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:58,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:58,435 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:58,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:58,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:58,536 DEBUG [Thread-1741] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:54:58,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:54:58,538 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:54:58,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:54:58,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:54:58,541 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:54:58,541 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:58,541 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:54:58,541 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:54:58,541 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:58,541 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:58,543 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:58,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:58,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:58,543 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:58,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:58,543 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:58,543 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:54:58,544 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:58,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:54:58,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:58,545 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:54:58,545 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:54:58,545 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:54:58,545 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:54:58,545 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:58,545 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:54:58,547 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:54:58,548 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:54:58,548 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:54:58,548 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. started... 2023-03-29 03:54:58,548 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2ed55fa255954e1310f6b9fd96e28980 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-29 03:54:58,548 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ed55fa255954e1310f6b9fd96e28980, store=A 2023-03-29 03:54:58,548 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:54:58,548 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ed55fa255954e1310f6b9fd96e28980, store=B 2023-03-29 03:54:58,548 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:54:58,548 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ed55fa255954e1310f6b9fd96e28980, store=C 2023-03-29 03:54:58,548 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:54:58,558 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=400 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/A/31dbac4d1e9a4fea893ebbe1edd899dc 2023-03-29 03:54:58,572 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=400 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/B/5c5f03cf6e2a49a1a95cec2c04ccee37 2023-03-29 03:54:58,586 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=400 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/C/57dbfc8899bd4b4197ff042894dac7f7 2023-03-29 03:54:58,589 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/A/31dbac4d1e9a4fea893ebbe1edd899dc as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/31dbac4d1e9a4fea893ebbe1edd899dc 2023-03-29 03:54:58,595 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/31dbac4d1e9a4fea893ebbe1edd899dc, entries=150, sequenceid=400, filesize=11.9 K 2023-03-29 03:54:58,600 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/B/5c5f03cf6e2a49a1a95cec2c04ccee37 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/5c5f03cf6e2a49a1a95cec2c04ccee37 2023-03-29 03:54:58,604 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/5c5f03cf6e2a49a1a95cec2c04ccee37, entries=150, sequenceid=400, filesize=11.9 K 2023-03-29 03:54:58,605 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/C/57dbfc8899bd4b4197ff042894dac7f7 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/57dbfc8899bd4b4197ff042894dac7f7 2023-03-29 03:54:58,609 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/57dbfc8899bd4b4197ff042894dac7f7, entries=150, sequenceid=400, filesize=11.9 K 2023-03-29 03:54:58,610 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=0 B/0 for 2ed55fa255954e1310f6b9fd96e28980 in 62ms, sequenceid=400, compaction requested=true 2023-03-29 03:54:58,610 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2ed55fa255954e1310f6b9fd96e28980: 2023-03-29 03:54:58,610 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:54:58,610 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:54:58,610 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:54:58,610 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:58,610 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:54:58,610 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:54:58,612 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:58,612 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:58,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:58,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:58,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:58,612 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:58,612 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:54:58,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:58,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:58,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:58,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:58,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:58,614 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:54:58,614 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@436c0e1b[Count = 0] remaining members to acquire global barrier 2023-03-29 03:54:58,614 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:54:58,614 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:58,616 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:58,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:58,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:58,616 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:54:58,616 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:54:58,616 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:54:58,616 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:58,616 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:54:58,617 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:54:58,617 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:58,617 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:58,617 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:54:58,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:58,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:58,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:58,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:58,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:58,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:58,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:58,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:58,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:58,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:58,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:54:58,621 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:54:58,621 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:54:58,621 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:54:58,621 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:54:58,621 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:54:58,621 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:54:58,626 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:58,626 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:58,626 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:58,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:58,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:58,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:58,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:58,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:58,626 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:58,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:58,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:58,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:58,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:58,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:58,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:58,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:58,627 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:58,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:58,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:58,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:58,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:58,631 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:58,631 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:58,631 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:58,631 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:58,631 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:58,631 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:54:58,631 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:54:58,631 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:58,631 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:58,631 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:58,631 DEBUG [Thread-1741] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:54:58,631 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:54:58,631 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:58,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:58,631 DEBUG [Thread-1741] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:54:58,631 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:58,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:58,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:58,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:58,732 DEBUG [Thread-1741] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:54:58,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:54:58,735 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:54:58,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:54:58,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:54:58,736 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:54:58,736 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:58,736 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:54:58,736 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:54:58,736 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:58,736 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:58,737 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:58,738 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:58,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:58,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:58,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:58,738 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:58,738 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:54:58,738 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:58,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:54:58,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:58,739 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:54:58,739 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:54:58,739 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:54:58,740 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:54:58,740 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:58,740 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:54:58,740 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:54:58,740 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:54:58,740 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:54:58,744 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. started... 2023-03-29 03:54:58,744 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2ed55fa255954e1310f6b9fd96e28980: 2023-03-29 03:54:58,744 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:54:58,744 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:54:58,744 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:54:58,744 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:58,744 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:54:58,744 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:54:58,746 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:58,746 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:58,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:58,746 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:58,746 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:54:58,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:58,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:58,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:58,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:58,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:58,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:58,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:58,748 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:54:58,748 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@650d0a00[Count = 0] remaining members to acquire global barrier 2023-03-29 03:54:58,748 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:54:58,748 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:58,749 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:58,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:58,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:58,749 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:54:58,749 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:54:58,749 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:54:58,749 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:58,749 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:54:58,752 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:54:58,752 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:58,752 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:58,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:58,752 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:54:58,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:58,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:58,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:58,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:58,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:58,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:58,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:58,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:58,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:58,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:54:58,755 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:54:58,755 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:54:58,755 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:54:58,755 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:54:58,755 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:54:58,755 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:54:58,756 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:58,756 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:58,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:58,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:58,756 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:58,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:58,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:58,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:58,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:58,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:58,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:58,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:58,757 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:58,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:58,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:58,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:58,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:58,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:58,759 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:58,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:58,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:58,764 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:58,764 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:58,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:58,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:58,764 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:58,764 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:58,764 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:58,764 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:58,764 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:58,764 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:58,764 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:58,764 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:54:58,765 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:54:58,765 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:58,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:58,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:58,765 DEBUG [Thread-1741] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:54:58,765 DEBUG [Thread-1741] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:54:58,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on 2ed55fa255954e1310f6b9fd96e28980 2023-03-29 03:54:58,813 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2ed55fa255954e1310f6b9fd96e28980 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-29 03:54:58,814 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ed55fa255954e1310f6b9fd96e28980, store=A 2023-03-29 03:54:58,814 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:54:58,814 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ed55fa255954e1310f6b9fd96e28980, store=B 2023-03-29 03:54:58,814 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:54:58,814 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ed55fa255954e1310f6b9fd96e28980, store=C 2023-03-29 03:54:58,814 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:54:58,865 DEBUG [Thread-1741] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:54:58,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:54:58,867 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:54:58,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:54:58,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:54:58,868 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:54:58,868 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:58,868 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:54:58,868 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:54:58,868 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:58,868 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:58,870 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:58,870 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:58,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:58,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:58,870 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:58,870 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:54:58,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:58,870 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:58,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:54:58,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:58,871 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:54:58,871 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:54:58,871 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:54:58,871 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:54:58,871 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:58,871 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:54:58,871 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:54:58,871 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:54:58,871 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:54:58,871 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. started... 2023-03-29 03:54:58,871 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. as already flushing 2023-03-29 03:54:58,871 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:54:58,872 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:54:58,872 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:54:58,872 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:58,872 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:54:58,872 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:54:58,874 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:58,874 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:58,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:58,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:58,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:58,874 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:58,874 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:54:58,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:58,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:58,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:58,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:58,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:58,875 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:54:58,876 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c978a55[Count = 0] remaining members to acquire global barrier 2023-03-29 03:54:58,876 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:54:58,876 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:58,877 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:58,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:58,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:58,877 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:54:58,877 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:54:58,877 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:54:58,877 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:58,877 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:54:58,878 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:58,879 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:54:58,879 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:58,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:58,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:58,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:58,879 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:54:58,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:58,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:58,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:58,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:58,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:58,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:58,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:58,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:54:58,881 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:54:58,881 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:54:58,881 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:54:58,881 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:54:58,881 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:54:58,881 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:54:58,883 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:58,883 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:58,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:58,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:58,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:58,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:58,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:58,883 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:58,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:58,883 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:58,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:58,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:58,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:58,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:58,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:58,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:58,885 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:58,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:58,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:58,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:58,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:58,887 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:58,887 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680062098853, max timestamp=1680062098885Num uniques 859; 2023-03-29 03:54:58,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37634 deadline: 1680062158883, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:58,887 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680062098853, max timestamp=1680062098885Num uniques 859; 2023-03-29 03:54:58,887 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-29 03:54:58,887 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-29 03:54:58,887 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680062098853, max timestamp=1680062098885Num uniques 859; 2023-03-29 03:54:58,887 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:58,887 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:58,887 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:58,887 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:54:58,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:58,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:58,887 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-29 03:54:58,887 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:54:58,888 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:54:58,887 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:58,887 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:58,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:58,888 DEBUG [Thread-1741] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:54:58,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:58,888 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:58,888 DEBUG [Thread-1741] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:54:58,888 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:58,888 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:58,888 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:58,888 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:58,916 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:58,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37570 deadline: 1680062158913, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:58,919 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:58,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37610 deadline: 1680062158916, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:58,988 DEBUG [Thread-1741] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:54:58,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:54:58,989 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:58,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37634 deadline: 1680062158988, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:58,990 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:54:58,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:54:58,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:54:58,992 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:54:58,992 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:58,992 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:54:58,992 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:54:58,992 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:58,992 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:58,994 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:58,994 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:58,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:58,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:58,994 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:58,994 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:54:58,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:58,994 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:58,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:54:58,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:58,995 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:54:58,995 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:54:58,995 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:54:58,995 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:54:58,995 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:58,995 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:54:58,995 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:54:58,995 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:54:58,995 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:54:58,995 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. started... 2023-03-29 03:54:58,995 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. as already flushing 2023-03-29 03:54:58,995 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:54:58,996 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:54:58,996 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:54:58,996 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:58,996 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:54:58,996 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:54:58,997 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:58,997 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:58,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:58,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:58,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:58,998 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:58,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:58,998 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:54:58,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:58,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:58,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:58,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:58,999 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:54:58,999 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5703a977[Count = 0] remaining members to acquire global barrier 2023-03-29 03:54:58,999 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:54:58,999 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:59,000 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:59,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:59,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:59,001 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:54:59,001 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:54:59,001 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:59,001 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:54:59,001 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:54:59,003 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:59,003 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:54:59,003 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:59,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:59,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:59,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:59,003 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:54:59,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:59,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:59,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:59,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:59,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:59,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:59,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:59,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:54:59,006 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:54:59,006 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:54:59,006 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:54:59,006 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:54:59,006 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:54:59,006 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:54:59,007 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:59,007 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:59,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:59,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:59,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:59,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:59,007 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:59,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:59,008 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:59,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:59,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:59,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:59,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:59,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:59,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:59,009 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:59,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:59,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:59,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:59,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:59,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:59,014 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:59,014 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:59,014 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:59,014 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:59,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:59,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:59,014 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:59,014 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:54:59,014 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:54:59,014 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:59,015 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:59,015 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:59,015 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:59,015 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:59,015 DEBUG [Thread-1741] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:54:59,015 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:54:59,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:59,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:59,015 DEBUG [Thread-1741] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:54:59,018 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:59,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37570 deadline: 1680062159017, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:59,021 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:59,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37610 deadline: 1680062159020, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:59,115 DEBUG [Thread-1741] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:54:59,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:54:59,117 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:54:59,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:54:59,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:54:59,118 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:54:59,118 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:59,118 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:54:59,118 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:54:59,119 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:59,119 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:59,121 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:59,121 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:59,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:59,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:59,122 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:59,122 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:54:59,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:59,122 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:59,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:54:59,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:59,122 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:54:59,123 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:54:59,123 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:54:59,123 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:54:59,123 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:59,123 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:54:59,123 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:54:59,123 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:54:59,123 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:54:59,123 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. started... 2023-03-29 03:54:59,123 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. as already flushing 2023-03-29 03:54:59,124 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:54:59,124 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:54:59,124 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:54:59,124 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:59,124 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:54:59,124 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:54:59,125 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:59,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:59,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:59,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:59,125 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:59,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:59,126 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:59,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:59,126 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:54:59,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:59,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:59,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:59,127 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:54:59,127 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@743c795a[Count = 0] remaining members to acquire global barrier 2023-03-29 03:54:59,127 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:54:59,127 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:59,128 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:59,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:59,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:59,129 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:54:59,129 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:54:59,129 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:54:59,129 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:59,129 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:54:59,130 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:59,130 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:54:59,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:59,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:59,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:59,130 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:59,130 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:54:59,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:59,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:59,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:59,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:59,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:59,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:59,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:59,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:54:59,133 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:54:59,133 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:54:59,133 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:54:59,133 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:54:59,133 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:54:59,133 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:54:59,135 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:59,135 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:59,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:59,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:59,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:59,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:59,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:59,135 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:59,136 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:59,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:59,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:59,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:59,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:59,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:59,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:59,137 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:59,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:59,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:59,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:59,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:59,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:59,139 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:59,139 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:59,139 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:59,139 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:54:59,139 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:54:59,139 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:59,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:59,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:59,139 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:59,139 DEBUG [Thread-1741] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:54:59,139 DEBUG [Thread-1741] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:54:59,139 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:59,139 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:59,139 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:59,139 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:59,139 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:59,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:59,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:59,140 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:54:59,194 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:59,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37634 deadline: 1680062159191, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:59,222 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:59,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37570 deadline: 1680062159220, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:59,226 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:59,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37610 deadline: 1680062159223, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:59,228 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=412 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/A/55b33dcdf6c9410e9c965495e2a87d76 2023-03-29 03:54:59,239 DEBUG [Thread-1741] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:54:59,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:54:59,242 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:54:59,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:54:59,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:54:59,244 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:54:59,244 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:59,244 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:54:59,244 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:54:59,244 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:59,244 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:59,245 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:59,245 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:59,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:59,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:59,246 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:59,246 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:54:59,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:59,246 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:59,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:54:59,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:59,247 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:54:59,247 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:54:59,247 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:54:59,247 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:54:59,247 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:59,247 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:54:59,248 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:54:59,248 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:54:59,248 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:54:59,248 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. started... 2023-03-29 03:54:59,248 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. as already flushing 2023-03-29 03:54:59,248 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:54:59,248 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:54:59,248 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:54:59,248 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:59,248 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:54:59,248 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:54:59,249 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:59,249 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:59,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:59,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:59,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:59,250 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:59,250 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:54:59,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:59,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:59,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:59,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:59,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:59,252 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:54:59,252 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@79ec30d7[Count = 0] remaining members to acquire global barrier 2023-03-29 03:54:59,252 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:54:59,252 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:59,253 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:59,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:59,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:59,253 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:54:59,253 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:54:59,254 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:54:59,254 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:59,254 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:54:59,256 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:54:59,256 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:59,256 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:59,256 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:54:59,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:59,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:59,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:59,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:59,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:59,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:59,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:59,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:59,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:59,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:59,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:54:59,259 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:54:59,259 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:54:59,259 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:54:59,259 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:54:59,259 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:54:59,259 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:54:59,260 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:59,260 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:59,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:59,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:59,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:59,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:59,260 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:59,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:59,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:59,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:59,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:59,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:59,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:59,261 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:59,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:59,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:59,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:59,262 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:59,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:59,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:59,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:59,264 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:59,265 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:59,265 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:59,265 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:59,265 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:54:59,265 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:54:59,265 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:54:59,265 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:59,265 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:59,265 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:59,265 DEBUG [Thread-1741] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:54:59,265 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:59,265 DEBUG [Thread-1741] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:54:59,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:59,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:59,265 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:59,265 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:59,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:59,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:59,365 DEBUG [Thread-1741] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:54:59,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:54:59,367 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:54:59,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:54:59,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:54:59,369 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:54:59,369 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:59,369 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:54:59,369 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:54:59,369 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:59,369 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:59,370 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:59,370 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:59,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:59,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:59,371 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:59,371 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:54:59,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:59,371 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:59,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:54:59,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:59,371 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:54:59,372 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:54:59,372 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:54:59,372 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:54:59,372 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:59,372 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:54:59,372 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:54:59,372 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:54:59,372 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:54:59,372 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. started... 2023-03-29 03:54:59,372 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. as already flushing 2023-03-29 03:54:59,372 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:54:59,372 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:54:59,372 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:54:59,372 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:59,372 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:54:59,373 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:54:59,375 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:59,375 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:59,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:59,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:59,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:59,375 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:59,375 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:54:59,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:59,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:59,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:59,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:59,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:59,376 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:54:59,376 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:54:59,376 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@39e9d89d[Count = 0] remaining members to acquire global barrier 2023-03-29 03:54:59,376 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:59,377 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:59,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:59,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:59,378 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:59,378 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:54:59,378 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:54:59,378 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:54:59,378 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:54:59,379 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:59,379 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:54:59,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:59,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:59,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:59,379 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:59,379 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:54:59,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:59,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:59,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:59,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:59,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:59,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:59,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:59,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:54:59,382 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:54:59,382 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:54:59,382 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:54:59,382 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:54:59,382 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:54:59,382 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:54:59,383 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:59,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:59,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:59,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:59,383 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:59,383 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:59,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:59,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:59,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:59,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:59,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:59,384 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:59,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:59,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:59,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:59,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:59,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:59,385 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:59,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:59,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:59,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:59,388 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:59,388 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:59,388 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:59,388 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:59,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:59,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:59,388 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:59,388 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:54:59,388 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:54:59,388 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:59,388 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:59,389 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:59,389 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:59,389 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:59,389 DEBUG [Thread-1741] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:54:59,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:59,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:59,389 DEBUG [Thread-1741] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:54:59,425 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:59,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37572 deadline: 1680062159423, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:59,426 DEBUG [Thread-1733] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4170 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980., hostname=jenkins-hbase4.apache.org,35599,1680062018534, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-29 03:54:59,489 DEBUG [Thread-1741] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:54:59,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:54:59,491 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:54:59,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:54:59,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:54:59,492 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:54:59,492 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:59,492 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:54:59,492 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:54:59,493 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:59,493 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:59,497 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:59,497 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:59,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:59,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:59,497 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:59,498 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:54:59,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:59,498 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:59,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:54:59,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:59,498 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:54:59,499 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:54:59,499 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:54:59,499 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:54:59,499 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:59,499 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:54:59,499 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:54:59,499 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:54:59,499 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:54:59,499 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. started... 2023-03-29 03:54:59,499 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. as already flushing 2023-03-29 03:54:59,499 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:54:59,499 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:59,499 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:54:59,500 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:54:59,500 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:59,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37634 deadline: 1680062159497, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:59,500 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:54:59,500 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:54:59,501 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:59,501 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:59,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:59,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:59,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:59,502 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:59,502 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:54:59,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:59,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:59,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:59,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:59,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:59,503 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:54:59,503 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@736b96c6[Count = 0] remaining members to acquire global barrier 2023-03-29 03:54:59,503 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:54:59,504 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:59,506 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:59,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:59,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:59,506 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:54:59,506 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:54:59,506 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:59,506 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:54:59,506 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:54:59,507 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:59,507 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:54:59,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:59,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:59,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:59,507 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:59,508 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:54:59,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:59,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:59,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:59,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:59,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:59,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:59,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:59,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:54:59,510 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:54:59,510 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:54:59,510 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:54:59,510 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:54:59,510 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:54:59,510 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:54:59,511 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:59,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:59,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:59,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:59,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:59,512 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:59,512 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:59,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:59,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:59,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:59,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:59,512 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:59,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:59,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:59,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:59,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:59,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:59,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:59,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:59,514 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:59,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:59,516 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:59,516 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:59,516 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:59,516 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:59,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:59,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:59,516 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:59,516 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:59,516 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:59,516 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:59,516 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:59,516 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:59,516 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:54:59,516 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:54:59,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:59,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:59,517 DEBUG [Thread-1741] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:54:59,517 DEBUG [Thread-1741] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:54:59,526 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:59,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37570 deadline: 1680062159524, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:59,529 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:59,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37610 deadline: 1680062159528, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:59,617 DEBUG [Thread-1741] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:54:59,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:54:59,619 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:54:59,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:54:59,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:54:59,620 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:54:59,620 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:59,620 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:54:59,620 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:54:59,620 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:59,621 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:59,622 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:59,622 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:59,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:59,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:59,623 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:59,623 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:54:59,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:59,623 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:59,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:54:59,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:59,624 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:54:59,624 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:54:59,624 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:54:59,624 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:54:59,624 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:59,624 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:54:59,624 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:54:59,624 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:54:59,624 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:54:59,624 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. started... 2023-03-29 03:54:59,624 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. as already flushing 2023-03-29 03:54:59,624 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:54:59,624 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:54:59,624 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:54:59,625 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:59,625 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:54:59,625 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:54:59,631 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:59,631 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:59,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:59,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:59,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:59,631 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:59,631 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:54:59,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:59,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:59,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:59,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:59,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:59,633 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:54:59,633 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@648ac758[Count = 0] remaining members to acquire global barrier 2023-03-29 03:54:59,633 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:54:59,633 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:59,634 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:59,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:59,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:59,634 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:54:59,634 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:54:59,634 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:54:59,634 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:59,634 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:54:59,635 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:54:59,635 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:59,635 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:59,635 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:54:59,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:59,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:59,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:59,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:59,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:59,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:59,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:59,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:59,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:59,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:59,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:54:59,638 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:54:59,638 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:54:59,638 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:54:59,638 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:54:59,638 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:54:59,638 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:54:59,640 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:59,640 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:59,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:59,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:54:59,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:54:59,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:59,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:59,641 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:59,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:59,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:59,641 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:59,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:59,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:54:59,642 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=412 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/B/2173a9cdc57b46699006a7fdb8bef655 2023-03-29 03:54:59,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:59,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:54:59,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:59,643 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:59,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:59,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:54:59,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:54:59,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:59,648 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:59,648 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:59,648 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:59,648 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:54:59,648 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:54:59,649 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:54:59,648 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:59,649 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:59,649 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:54:59,649 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:59,649 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:59,649 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:54:59,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:59,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:59,649 DEBUG [Thread-1741] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:54:59,649 DEBUG [Thread-1741] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:54:59,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:54:59,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:54:59,664 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=412 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/C/867eb19dc0d34e5a8c33f0d2cd4e637f 2023-03-29 03:54:59,670 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:54:59,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37614 deadline: 1680062159666, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:59,673 DEBUG [Thread-1739] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8172 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980., hostname=jenkins-hbase4.apache.org,35599,1680062018534, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-29 03:54:59,677 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/A/55b33dcdf6c9410e9c965495e2a87d76 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/55b33dcdf6c9410e9c965495e2a87d76 2023-03-29 03:54:59,682 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/55b33dcdf6c9410e9c965495e2a87d76, entries=150, sequenceid=412, filesize=11.9 K 2023-03-29 03:54:59,683 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/B/2173a9cdc57b46699006a7fdb8bef655 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/2173a9cdc57b46699006a7fdb8bef655 2023-03-29 03:54:59,698 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/2173a9cdc57b46699006a7fdb8bef655, entries=150, sequenceid=412, filesize=11.9 K 2023-03-29 03:54:59,698 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/C/867eb19dc0d34e5a8c33f0d2cd4e637f as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/867eb19dc0d34e5a8c33f0d2cd4e637f 2023-03-29 03:54:59,707 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/867eb19dc0d34e5a8c33f0d2cd4e637f, entries=150, sequenceid=412, filesize=11.9 K 2023-03-29 03:54:59,708 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 2ed55fa255954e1310f6b9fd96e28980 in 894ms, sequenceid=412, compaction requested=true 2023-03-29 03:54:59,708 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2ed55fa255954e1310f6b9fd96e28980: 2023-03-29 03:54:59,708 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:54:59,708 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-29 03:54:59,708 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:54:59,708 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-29 03:54:59,708 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-29 03:54:59,709 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49630 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-29 03:54:59,710 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1912): 2ed55fa255954e1310f6b9fd96e28980/B is initiating minor compaction (all files) 2023-03-29 03:54:59,710 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2ed55fa255954e1310f6b9fd96e28980/B in TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:54:59,710 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/1b6c5a6406044890b8993cc36b060672, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/b23952d5900849b693e69ff429be80c3, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/5c5f03cf6e2a49a1a95cec2c04ccee37, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/2173a9cdc57b46699006a7fdb8bef655] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp, totalSize=48.5 K 2023-03-29 03:54:59,710 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 1b6c5a6406044890b8993cc36b060672, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=351, earliestPutTs=1680062097012 2023-03-29 03:54:59,711 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting b23952d5900849b693e69ff429be80c3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=386, earliestPutTs=1680062097065 2023-03-29 03:54:59,711 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 5c5f03cf6e2a49a1a95cec2c04ccee37, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=400, earliestPutTs=1680062097684 2023-03-29 03:54:59,711 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 2173a9cdc57b46699006a7fdb8bef655, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=412, earliestPutTs=1680062098806 2023-03-29 03:54:59,721 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] throttle.PressureAwareThroughputController(145): 2ed55fa255954e1310f6b9fd96e28980#B#compaction#310 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:54:59,723 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49630 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-29 03:54:59,724 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1912): 2ed55fa255954e1310f6b9fd96e28980/A is initiating minor compaction (all files) 2023-03-29 03:54:59,724 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2ed55fa255954e1310f6b9fd96e28980/A in TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:54:59,724 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/98e78e88fa334457a10d8d8bafcc31ac, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/932e0c76313543b29e70d1b8be69fff8, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/31dbac4d1e9a4fea893ebbe1edd899dc, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/55b33dcdf6c9410e9c965495e2a87d76] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp, totalSize=48.5 K 2023-03-29 03:54:59,724 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 98e78e88fa334457a10d8d8bafcc31ac, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=351, earliestPutTs=1680062097012 2023-03-29 03:54:59,725 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 932e0c76313543b29e70d1b8be69fff8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=386, earliestPutTs=1680062097065 2023-03-29 03:54:59,725 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 31dbac4d1e9a4fea893ebbe1edd899dc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=400, earliestPutTs=1680062097684 2023-03-29 03:54:59,725 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 55b33dcdf6c9410e9c965495e2a87d76, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=412, earliestPutTs=1680062098806 2023-03-29 03:54:59,749 DEBUG [Thread-1741] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:54:59,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:54:59,752 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:54:59,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:54:59,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:54:59,754 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2ed55fa255954e1310f6b9fd96e28980#A#compaction#311 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:54:59,754 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:54:59,754 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:59,754 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:54:59,754 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:54:59,754 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:59,754 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:59,756 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:54:59,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:54:59,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:54:59,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:59,757 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:54:59,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:54:59,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:54:59,758 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:54:59,758 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:59,758 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:54:59,758 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:54:59,758 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:54:59,759 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:54:59,759 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:54:59,759 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:54:59,759 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:54:59,759 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:54:59,759 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:54:59,759 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:54:59,759 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. started... 2023-03-29 03:54:59,759 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2ed55fa255954e1310f6b9fd96e28980 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-03-29 03:54:59,759 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ed55fa255954e1310f6b9fd96e28980, store=A 2023-03-29 03:54:59,759 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-29 03:54:59,760 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ed55fa255954e1310f6b9fd96e28980, store=B 2023-03-29 03:54:59,760 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-29 03:54:59,760 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ed55fa255954e1310f6b9fd96e28980, store=C 2023-03-29 03:54:59,760 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-29 03:54:59,763 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/B/976f7d2e5d8e45789a99f5d11c4b7ac7 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/976f7d2e5d8e45789a99f5d11c4b7ac7 2023-03-29 03:54:59,769 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 2ed55fa255954e1310f6b9fd96e28980/B of 2ed55fa255954e1310f6b9fd96e28980 into 976f7d2e5d8e45789a99f5d11c4b7ac7(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:54:59,769 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2ed55fa255954e1310f6b9fd96e28980: 2023-03-29 03:54:59,769 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980., storeName=2ed55fa255954e1310f6b9fd96e28980/B, priority=12, startTime=1680062099708; duration=0sec 2023-03-29 03:54:59,769 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:54:59,769 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-29 03:54:59,770 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49630 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-29 03:54:59,770 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1912): 2ed55fa255954e1310f6b9fd96e28980/C is initiating minor compaction (all files) 2023-03-29 03:54:59,770 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2ed55fa255954e1310f6b9fd96e28980/C in TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:54:59,770 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/78dd16f5f1c441d08f5100f41a11211e, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/7f5458004d224d53b183f0cca1bdf866, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/57dbfc8899bd4b4197ff042894dac7f7, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/867eb19dc0d34e5a8c33f0d2cd4e637f] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp, totalSize=48.5 K 2023-03-29 03:54:59,771 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 78dd16f5f1c441d08f5100f41a11211e, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=351, earliestPutTs=1680062097012 2023-03-29 03:54:59,771 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 7f5458004d224d53b183f0cca1bdf866, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=386, earliestPutTs=1680062097065 2023-03-29 03:54:59,771 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 57dbfc8899bd4b4197ff042894dac7f7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=400, earliestPutTs=1680062097684 2023-03-29 03:54:59,772 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 867eb19dc0d34e5a8c33f0d2cd4e637f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=412, earliestPutTs=1680062098806 2023-03-29 03:54:59,785 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] throttle.PressureAwareThroughputController(145): 2ed55fa255954e1310f6b9fd96e28980#C#compaction#313 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:54:59,794 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/C/00dec3b578e14a1db005e73ec33c4d14 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/00dec3b578e14a1db005e73ec33c4d14 2023-03-29 03:54:59,799 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 2ed55fa255954e1310f6b9fd96e28980/C of 2ed55fa255954e1310f6b9fd96e28980 into 00dec3b578e14a1db005e73ec33c4d14(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:54:59,799 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2ed55fa255954e1310f6b9fd96e28980: 2023-03-29 03:54:59,799 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980., storeName=2ed55fa255954e1310f6b9fd96e28980/C, priority=12, startTime=1680062099708; duration=0sec 2023-03-29 03:54:59,800 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:55:00,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on 2ed55fa255954e1310f6b9fd96e28980 2023-03-29 03:55:00,006 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. as already flushing 2023-03-29 03:55:00,026 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:00,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37634 deadline: 1680062160025, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:00,032 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:00,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37570 deadline: 1680062160030, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:00,036 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:00,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37610 deadline: 1680062160034, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:00,129 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:00,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37634 deadline: 1680062160127, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:00,165 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/A/7fe8a3f221f7463bae715f263264e4f3 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/7fe8a3f221f7463bae715f263264e4f3 2023-03-29 03:55:00,169 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 2ed55fa255954e1310f6b9fd96e28980/A of 2ed55fa255954e1310f6b9fd96e28980 into 7fe8a3f221f7463bae715f263264e4f3(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:55:00,169 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2ed55fa255954e1310f6b9fd96e28980: 2023-03-29 03:55:00,169 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980., storeName=2ed55fa255954e1310f6b9fd96e28980/A, priority=12, startTime=1680062099708; duration=0sec 2023-03-29 03:55:00,169 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:55:00,179 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=448 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/A/76468a509c1048e68ef9763c0b34c84d 2023-03-29 03:55:00,189 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=448 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/B/9582ecfeab924bfbb38b5b4e0d40aa3e 2023-03-29 03:55:00,333 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:00,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37634 deadline: 1680062160332, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:00,598 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=448 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/C/186760cc625d4bffb865ec589aea5142 2023-03-29 03:55:00,602 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/A/76468a509c1048e68ef9763c0b34c84d as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/76468a509c1048e68ef9763c0b34c84d 2023-03-29 03:55:00,606 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/76468a509c1048e68ef9763c0b34c84d, entries=150, sequenceid=448, filesize=11.9 K 2023-03-29 03:55:00,606 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/B/9582ecfeab924bfbb38b5b4e0d40aa3e as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/9582ecfeab924bfbb38b5b4e0d40aa3e 2023-03-29 03:55:00,610 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/9582ecfeab924bfbb38b5b4e0d40aa3e, entries=150, sequenceid=448, filesize=11.9 K 2023-03-29 03:55:00,610 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/C/186760cc625d4bffb865ec589aea5142 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/186760cc625d4bffb865ec589aea5142 2023-03-29 03:55:00,614 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/186760cc625d4bffb865ec589aea5142, entries=150, sequenceid=448, filesize=11.9 K 2023-03-29 03:55:00,614 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=53.67 KB/54960 for 2ed55fa255954e1310f6b9fd96e28980 in 855ms, sequenceid=448, compaction requested=false 2023-03-29 03:55:00,615 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2ed55fa255954e1310f6b9fd96e28980: 2023-03-29 03:55:00,615 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:55:00,615 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:00,615 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:00,615 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:00,615 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:00,615 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:00,616 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:00,616 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:00,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:00,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:00,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:00,617 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:00,617 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:00,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:00,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:00,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:00,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:00,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:00,618 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:00,618 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7cf82996[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:00,618 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:00,618 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:00,620 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:00,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:00,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:00,620 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:00,620 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:00,620 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:00,621 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:00,621 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:00,622 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:00,622 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:00,622 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:00,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:00,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:00,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:00,622 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:00,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:00,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:00,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:00,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:00,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:00,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:00,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:00,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:00,624 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:00,624 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:00,624 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:00,624 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:00,624 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:00,624 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:00,625 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:00,625 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:00,626 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:00,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:00,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:00,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:00,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:00,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:00,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:00,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:00,626 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:00,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:00,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:00,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:00,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:00,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:00,627 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:00,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:00,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:00,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:00,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:00,630 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:00,630 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:00,630 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:00,630 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:00,630 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:00,630 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:00,630 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:00,630 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:55:00,630 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:00,630 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:00,630 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:00,630 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:00,630 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:00,630 DEBUG [Thread-1741] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:00,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:00,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:00,630 DEBUG [Thread-1741] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:00,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:00,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:00,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on 2ed55fa255954e1310f6b9fd96e28980 2023-03-29 03:55:00,637 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2ed55fa255954e1310f6b9fd96e28980 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-29 03:55:00,637 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ed55fa255954e1310f6b9fd96e28980, store=A 2023-03-29 03:55:00,637 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:55:00,637 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ed55fa255954e1310f6b9fd96e28980, store=B 2023-03-29 03:55:00,637 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:55:00,637 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ed55fa255954e1310f6b9fd96e28980, store=C 2023-03-29 03:55:00,637 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:55:00,644 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=463 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/A/30195084e4e34157ae842f3c81370e78 2023-03-29 03:55:00,687 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:00,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37634 deadline: 1680062160686, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:00,730 DEBUG [Thread-1741] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:00,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:00,733 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:00,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:00,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:00,734 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:00,734 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:00,734 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:00,734 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:00,734 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:00,734 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:00,736 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:00,736 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:00,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:00,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:00,736 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:00,736 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:00,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:00,737 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:00,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:00,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:00,737 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:00,737 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:00,737 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:00,737 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:00,737 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:00,737 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:00,738 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:00,738 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:00,738 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:55:00,738 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. started... 2023-03-29 03:55:00,738 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. as already flushing 2023-03-29 03:55:00,738 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:55:00,738 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:00,738 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:00,738 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:00,738 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:00,738 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:00,744 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:00,744 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:00,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:00,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:00,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:00,744 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:00,744 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:00,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:00,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:00,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:00,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:00,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:00,746 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:00,746 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b223ba4[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:00,746 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:00,746 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:00,747 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:00,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:00,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:00,747 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:00,747 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:00,747 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:00,747 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:00,747 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:00,748 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:00,749 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:00,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:00,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:00,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:00,749 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:00,749 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:00,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:00,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:00,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:00,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:00,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:00,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:00,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:00,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:00,751 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:00,751 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:00,751 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:00,751 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:00,751 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:00,751 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:00,754 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:00,754 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:00,754 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:00,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:00,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:00,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:00,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:00,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:00,754 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:00,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:00,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:00,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:00,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:00,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:00,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:00,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:00,755 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:00,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:00,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:00,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:00,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:00,758 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:00,758 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:00,758 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:00,758 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:00,758 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:00,758 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:00,758 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:00,758 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:00,758 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:00,758 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:55:00,758 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:00,758 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:00,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:00,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:00,758 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:00,758 DEBUG [Thread-1741] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:00,759 DEBUG [Thread-1741] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:00,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:00,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:00,790 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:00,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 246 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37634 deadline: 1680062160788, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:00,859 DEBUG [Thread-1741] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:00,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:00,861 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:00,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:00,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:00,863 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:00,863 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:00,863 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:00,863 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:00,863 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:00,864 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:00,865 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:00,865 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:00,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:00,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:00,865 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:00,865 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:00,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:00,866 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:00,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:00,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:00,866 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:00,871 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:00,872 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:00,872 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:00,872 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:00,872 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:00,872 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:00,872 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:00,872 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:55:00,878 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. started... 2023-03-29 03:55:00,878 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. as already flushing 2023-03-29 03:55:00,878 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:55:00,878 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:00,878 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:00,878 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:00,878 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:00,878 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:00,880 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:00,880 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:00,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:00,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:00,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:00,881 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:00,881 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:00,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:00,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:00,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:00,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:00,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:00,882 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:00,882 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@24d051a0[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:00,882 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:00,882 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:00,884 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:00,884 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:00,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:00,884 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:00,884 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:00,884 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:00,885 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:00,885 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:00,886 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:00,886 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:00,886 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:00,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:00,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:00,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:00,886 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:00,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:00,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:00,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:00,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:00,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:00,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:00,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:00,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:00,889 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:00,889 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:00,889 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:00,889 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:00,890 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:00,890 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:00,891 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:00,891 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:00,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:00,891 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:00,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:00,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:00,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:00,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:00,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:00,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:00,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:00,892 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:00,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:00,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:00,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:00,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:00,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:00,894 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:00,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:00,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:00,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:00,896 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:00,896 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:00,896 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:00,896 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:00,896 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:00,896 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:00,896 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:00,896 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:00,896 DEBUG [Thread-1741] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:00,897 DEBUG [Thread-1741] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:00,896 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:00,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:00,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:00,897 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:00,897 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:00,897 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:00,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:00,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:00,993 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:00,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 248 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37634 deadline: 1680062160992, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:00,997 DEBUG [Thread-1741] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:00,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:00,999 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:01,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:01,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:01,000 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:01,000 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:01,000 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:01,000 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:01,000 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:01,000 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:01,002 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:01,002 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:01,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:01,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:01,002 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:01,003 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:01,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:01,003 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:01,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:01,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:01,003 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:01,004 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:01,004 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:01,004 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:01,004 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:01,004 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:01,004 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:01,004 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:01,004 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:55:01,004 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. started... 2023-03-29 03:55:01,004 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. as already flushing 2023-03-29 03:55:01,004 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:55:01,004 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:01,004 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:01,004 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:01,004 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:01,004 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:01,006 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:01,006 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:01,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:01,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:01,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:01,006 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:01,006 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:01,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:01,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:01,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:01,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:01,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:01,008 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:01,008 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1652f265[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:01,008 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:01,008 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:01,009 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:01,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:01,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:01,009 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:01,009 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:01,009 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:01,009 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:01,009 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:01,011 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:01,011 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:01,011 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:01,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:01,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:01,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:01,011 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:01,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:01,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:01,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:01,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:01,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:01,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:01,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:01,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:01,013 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:01,013 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:01,013 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:01,013 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:01,013 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:01,013 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:01,016 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:01,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:01,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:01,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:01,016 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:01,016 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:01,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:01,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:01,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:01,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:01,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:01,017 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:01,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:01,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:01,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:01,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:01,018 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:01,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:01,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:01,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:01,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:01,021 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:01,021 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:01,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:01,021 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:01,021 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:01,021 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:01,021 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:01,021 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:01,021 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:01,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:01,022 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:01,022 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:01,022 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:01,022 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:01,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:01,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:01,022 DEBUG [Thread-1741] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:01,022 DEBUG [Thread-1741] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:01,037 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:01,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37570 deadline: 1680062161036, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:01,046 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:01,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37610 deadline: 1680062161045, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:01,053 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=463 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/B/498b866acfdd45689752d73d71e8b36b 2023-03-29 03:55:01,063 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=463 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/C/cb956507fe85419e868d7c6e5b73d4a2 2023-03-29 03:55:01,066 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/A/30195084e4e34157ae842f3c81370e78 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/30195084e4e34157ae842f3c81370e78 2023-03-29 03:55:01,076 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/30195084e4e34157ae842f3c81370e78, entries=200, sequenceid=463, filesize=14.3 K 2023-03-29 03:55:01,077 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/B/498b866acfdd45689752d73d71e8b36b as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/498b866acfdd45689752d73d71e8b36b 2023-03-29 03:55:01,080 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/498b866acfdd45689752d73d71e8b36b, entries=150, sequenceid=463, filesize=11.9 K 2023-03-29 03:55:01,081 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/C/cb956507fe85419e868d7c6e5b73d4a2 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/cb956507fe85419e868d7c6e5b73d4a2 2023-03-29 03:55:01,084 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/cb956507fe85419e868d7c6e5b73d4a2, entries=150, sequenceid=463, filesize=11.9 K 2023-03-29 03:55:01,085 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 2ed55fa255954e1310f6b9fd96e28980 in 448ms, sequenceid=463, compaction requested=true 2023-03-29 03:55:01,085 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2ed55fa255954e1310f6b9fd96e28980: 2023-03-29 03:55:01,085 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:55:01,085 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-29 03:55:01,085 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-29 03:55:01,085 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:55:01,085 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:55:01,086 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 40020 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-29 03:55:01,086 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37580 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-29 03:55:01,086 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1912): 2ed55fa255954e1310f6b9fd96e28980/B is initiating minor compaction (all files) 2023-03-29 03:55:01,086 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1912): 2ed55fa255954e1310f6b9fd96e28980/A is initiating minor compaction (all files) 2023-03-29 03:55:01,086 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2ed55fa255954e1310f6b9fd96e28980/B in TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:55:01,086 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2ed55fa255954e1310f6b9fd96e28980/A in TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:55:01,086 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/976f7d2e5d8e45789a99f5d11c4b7ac7, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/9582ecfeab924bfbb38b5b4e0d40aa3e, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/498b866acfdd45689752d73d71e8b36b] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp, totalSize=36.7 K 2023-03-29 03:55:01,086 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/7fe8a3f221f7463bae715f263264e4f3, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/76468a509c1048e68ef9763c0b34c84d, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/30195084e4e34157ae842f3c81370e78] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp, totalSize=39.1 K 2023-03-29 03:55:01,087 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 976f7d2e5d8e45789a99f5d11c4b7ac7, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=412, earliestPutTs=1680062098806 2023-03-29 03:55:01,087 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 7fe8a3f221f7463bae715f263264e4f3, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=412, earliestPutTs=1680062098806 2023-03-29 03:55:01,087 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 9582ecfeab924bfbb38b5b4e0d40aa3e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=448, earliestPutTs=1680062098908 2023-03-29 03:55:01,087 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 76468a509c1048e68ef9763c0b34c84d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=448, earliestPutTs=1680062098908 2023-03-29 03:55:01,087 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 498b866acfdd45689752d73d71e8b36b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=463, earliestPutTs=1680062100010 2023-03-29 03:55:01,088 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 30195084e4e34157ae842f3c81370e78, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=463, earliestPutTs=1680062100010 2023-03-29 03:55:01,094 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2ed55fa255954e1310f6b9fd96e28980#A#compaction#320 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:55:01,094 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] throttle.PressureAwareThroughputController(145): 2ed55fa255954e1310f6b9fd96e28980#B#compaction#319 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:55:01,108 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/A/ec74989f9707496ca776d9daaa193619 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/ec74989f9707496ca776d9daaa193619 2023-03-29 03:55:01,112 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2ed55fa255954e1310f6b9fd96e28980/A of 2ed55fa255954e1310f6b9fd96e28980 into ec74989f9707496ca776d9daaa193619(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:55:01,112 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2ed55fa255954e1310f6b9fd96e28980: 2023-03-29 03:55:01,112 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980., storeName=2ed55fa255954e1310f6b9fd96e28980/A, priority=13, startTime=1680062101085; duration=0sec 2023-03-29 03:55:01,112 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:55:01,112 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-29 03:55:01,113 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37580 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-29 03:55:01,113 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1912): 2ed55fa255954e1310f6b9fd96e28980/C is initiating minor compaction (all files) 2023-03-29 03:55:01,113 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2ed55fa255954e1310f6b9fd96e28980/C in TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:55:01,114 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/00dec3b578e14a1db005e73ec33c4d14, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/186760cc625d4bffb865ec589aea5142, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/cb956507fe85419e868d7c6e5b73d4a2] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp, totalSize=36.7 K 2023-03-29 03:55:01,114 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 00dec3b578e14a1db005e73ec33c4d14, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=412, earliestPutTs=1680062098806 2023-03-29 03:55:01,114 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 186760cc625d4bffb865ec589aea5142, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=448, earliestPutTs=1680062098908 2023-03-29 03:55:01,114 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting cb956507fe85419e868d7c6e5b73d4a2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=463, earliestPutTs=1680062100010 2023-03-29 03:55:01,119 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2ed55fa255954e1310f6b9fd96e28980#C#compaction#321 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:55:01,122 DEBUG [Thread-1741] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:01,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:01,124 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:01,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:01,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:01,125 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:01,125 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:01,125 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:01,125 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:01,126 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:01,126 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:01,127 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:01,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:01,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:01,128 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:01,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:01,128 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:01,128 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:01,128 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:01,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:01,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:01,129 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:01,129 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:01,129 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:01,129 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:01,129 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:01,129 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:01,129 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:01,129 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:01,129 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:55:01,129 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. started... 2023-03-29 03:55:01,129 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2ed55fa255954e1310f6b9fd96e28980 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-29 03:55:01,130 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ed55fa255954e1310f6b9fd96e28980, store=A 2023-03-29 03:55:01,130 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:55:01,130 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ed55fa255954e1310f6b9fd96e28980, store=B 2023-03-29 03:55:01,130 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:55:01,130 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ed55fa255954e1310f6b9fd96e28980, store=C 2023-03-29 03:55:01,130 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:55:01,130 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/C/f94c17bd8cf24fbf82f3a8e9486bac34 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/f94c17bd8cf24fbf82f3a8e9486bac34 2023-03-29 03:55:01,135 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2ed55fa255954e1310f6b9fd96e28980/C of 2ed55fa255954e1310f6b9fd96e28980 into f94c17bd8cf24fbf82f3a8e9486bac34(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:55:01,135 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2ed55fa255954e1310f6b9fd96e28980: 2023-03-29 03:55:01,135 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980., storeName=2ed55fa255954e1310f6b9fd96e28980/C, priority=13, startTime=1680062101085; duration=0sec 2023-03-29 03:55:01,135 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:55:01,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on 2ed55fa255954e1310f6b9fd96e28980 2023-03-29 03:55:01,297 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. as already flushing 2023-03-29 03:55:01,318 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:01,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 259 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37634 deadline: 1680062161318, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:01,421 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:01,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 261 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37634 deadline: 1680062161419, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:01,506 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/B/d3631fed20f1402084bbe0aba71373ff as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/d3631fed20f1402084bbe0aba71373ff 2023-03-29 03:55:01,509 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2ed55fa255954e1310f6b9fd96e28980/B of 2ed55fa255954e1310f6b9fd96e28980 into d3631fed20f1402084bbe0aba71373ff(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:55:01,510 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2ed55fa255954e1310f6b9fd96e28980: 2023-03-29 03:55:01,510 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980., storeName=2ed55fa255954e1310f6b9fd96e28980/B, priority=13, startTime=1680062101085; duration=0sec 2023-03-29 03:55:01,510 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:55:01,537 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=484 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/A/b410dc1cfd2043d0abbe715abeb5437d 2023-03-29 03:55:01,624 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:01,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 263 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37634 deadline: 1680062161623, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:01,927 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:01,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 265 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37634 deadline: 1680062161926, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:01,955 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=484 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/B/7cac2373fedb4932a4ca882e0779a51b 2023-03-29 03:55:01,974 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=484 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/C/fb284284b79f446d8f9c1da423f971b7 2023-03-29 03:55:01,978 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/A/b410dc1cfd2043d0abbe715abeb5437d as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/b410dc1cfd2043d0abbe715abeb5437d 2023-03-29 03:55:01,982 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/b410dc1cfd2043d0abbe715abeb5437d, entries=150, sequenceid=484, filesize=11.9 K 2023-03-29 03:55:01,983 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/B/7cac2373fedb4932a4ca882e0779a51b as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/7cac2373fedb4932a4ca882e0779a51b 2023-03-29 03:55:01,986 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/7cac2373fedb4932a4ca882e0779a51b, entries=150, sequenceid=484, filesize=11.9 K 2023-03-29 03:55:01,987 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/C/fb284284b79f446d8f9c1da423f971b7 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/fb284284b79f446d8f9c1da423f971b7 2023-03-29 03:55:01,990 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/fb284284b79f446d8f9c1da423f971b7, entries=150, sequenceid=484, filesize=11.9 K 2023-03-29 03:55:01,991 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 2ed55fa255954e1310f6b9fd96e28980 in 862ms, sequenceid=484, compaction requested=false 2023-03-29 03:55:01,991 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2ed55fa255954e1310f6b9fd96e28980: 2023-03-29 03:55:01,991 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:55:01,991 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:01,991 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:01,991 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:01,991 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:01,991 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:01,993 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:01,993 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:01,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:01,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:01,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:01,993 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:01,993 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:01,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:01,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:01,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:01,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:01,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:01,995 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:01,995 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b5bcda0[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:01,995 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:01,995 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:01,997 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:01,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:01,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:01,997 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:01,997 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:01,997 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:01,998 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:01,998 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:02,006 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:02,006 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:02,006 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:02,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:02,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:02,007 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:02,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:02,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:02,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:02,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:02,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:02,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:02,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:02,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:02,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:02,010 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:02,010 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:02,010 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:02,010 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:02,010 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:02,010 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:02,012 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:02,012 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:02,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:02,012 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:02,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:02,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:02,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:02,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:02,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:02,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:02,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:02,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:02,013 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:02,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:02,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:02,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:02,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:02,014 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:02,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:02,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:02,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:02,016 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:02,016 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:02,016 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:02,016 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:02,016 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:02,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:02,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:02,016 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:02,017 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:02,017 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:02,017 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:02,017 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:02,017 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:02,017 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:02,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:02,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:02,017 DEBUG [Thread-1741] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:02,017 DEBUG [Thread-1741] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:02,117 DEBUG [Thread-1741] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:02,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:02,119 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:02,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:02,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:02,121 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:02,121 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:02,121 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:02,121 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:02,121 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:02,121 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:02,124 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:02,124 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:02,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:02,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:02,125 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:02,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:02,125 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:02,125 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:02,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:02,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:02,126 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:02,126 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:02,126 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:02,126 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:02,126 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:02,126 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:02,126 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:02,126 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:02,126 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:55:02,126 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. started... 2023-03-29 03:55:02,127 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2ed55fa255954e1310f6b9fd96e28980 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-29 03:55:02,127 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ed55fa255954e1310f6b9fd96e28980, store=A 2023-03-29 03:55:02,127 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:55:02,127 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ed55fa255954e1310f6b9fd96e28980, store=B 2023-03-29 03:55:02,127 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:55:02,127 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ed55fa255954e1310f6b9fd96e28980, store=C 2023-03-29 03:55:02,127 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:55:02,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on 2ed55fa255954e1310f6b9fd96e28980 2023-03-29 03:55:02,433 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. as already flushing 2023-03-29 03:55:02,474 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:02,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 284 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37634 deadline: 1680062162472, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:02,547 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=498 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/A/3555d3174aa8401abfb4545673c78212 2023-03-29 03:55:02,558 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=498 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/B/5f12434a444547f48363f426f5376f51 2023-03-29 03:55:02,576 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:02,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 286 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37634 deadline: 1680062162575, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:02,778 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:02,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 288 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37634 deadline: 1680062162777, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:02,969 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=498 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/C/bc662949ab5e47eb80fc5b241600b255 2023-03-29 03:55:02,973 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/A/3555d3174aa8401abfb4545673c78212 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/3555d3174aa8401abfb4545673c78212 2023-03-29 03:55:02,976 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/3555d3174aa8401abfb4545673c78212, entries=150, sequenceid=498, filesize=11.9 K 2023-03-29 03:55:02,977 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/B/5f12434a444547f48363f426f5376f51 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/5f12434a444547f48363f426f5376f51 2023-03-29 03:55:02,981 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/5f12434a444547f48363f426f5376f51, entries=150, sequenceid=498, filesize=11.9 K 2023-03-29 03:55:02,981 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/C/bc662949ab5e47eb80fc5b241600b255 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/bc662949ab5e47eb80fc5b241600b255 2023-03-29 03:55:02,984 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/bc662949ab5e47eb80fc5b241600b255, entries=150, sequenceid=498, filesize=11.9 K 2023-03-29 03:55:02,985 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 2ed55fa255954e1310f6b9fd96e28980 in 858ms, sequenceid=498, compaction requested=true 2023-03-29 03:55:02,985 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2ed55fa255954e1310f6b9fd96e28980: 2023-03-29 03:55:02,985 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:55:02,985 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:02,985 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:02,985 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:02,985 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:02,985 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:02,987 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:02,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:02,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:02,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:02,987 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:02,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:02,988 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:02,988 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:02,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:02,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:02,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:02,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:02,989 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:02,989 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@313e2c74[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:02,989 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:02,989 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:02,990 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:02,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:02,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:02,990 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:02,990 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:02,990 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:02,990 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:02,990 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:02,993 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:02,993 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:02,993 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:02,993 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:02,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:02,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:02,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:02,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:02,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:02,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:02,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:02,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:02,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:02,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:02,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:02,996 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:02,996 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:02,996 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:02,996 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:02,996 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:02,996 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:02,997 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:02,997 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:02,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:02,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:02,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:02,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:02,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:02,998 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:02,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:02,998 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:02,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:02,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:02,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:02,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:02,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:02,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:02,999 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:02,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:02,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:03,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:03,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:03,001 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:03,001 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:03,001 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:03,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:03,001 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:03,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:03,001 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:03,002 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:03,001 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:03,001 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:03,002 DEBUG [Thread-1741] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:03,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:03,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:03,002 DEBUG [Thread-1741] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:03,002 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:03,002 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:03,002 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:03,002 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:03,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on 2ed55fa255954e1310f6b9fd96e28980 2023-03-29 03:55:03,056 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2ed55fa255954e1310f6b9fd96e28980 3/3 column families, dataSize=120.76 KB heapSize=360.05 KB 2023-03-29 03:55:03,056 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ed55fa255954e1310f6b9fd96e28980, store=A 2023-03-29 03:55:03,056 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-29 03:55:03,056 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ed55fa255954e1310f6b9fd96e28980, store=B 2023-03-29 03:55:03,057 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-29 03:55:03,057 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ed55fa255954e1310f6b9fd96e28980, store=C 2023-03-29 03:55:03,057 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-29 03:55:03,068 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:03,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37610 deadline: 1680062163067, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:03,071 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:03,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37570 deadline: 1680062163069, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:03,081 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:03,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 290 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37634 deadline: 1680062163079, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:03,102 DEBUG [Thread-1741] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:03,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:03,104 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:03,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:03,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:03,106 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:03,106 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:03,106 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:03,107 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:03,107 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:03,107 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:03,108 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:03,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:03,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:03,109 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:03,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:03,109 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:03,109 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:03,109 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:03,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:03,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:03,110 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:03,110 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:03,110 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:03,110 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:03,110 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:03,110 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:03,110 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:03,110 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:03,110 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:55:03,110 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. started... 2023-03-29 03:55:03,110 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. as already flushing 2023-03-29 03:55:03,110 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:55:03,110 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:03,110 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:03,110 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:03,111 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:03,111 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:03,113 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:03,113 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:03,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:03,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:03,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:03,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:03,113 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:03,113 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:03,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:03,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:03,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:03,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:03,115 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:03,115 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@174536a2[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:03,115 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:03,115 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:03,116 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:03,116 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:03,116 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:03,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:03,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:03,116 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:03,117 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:03,117 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:03,118 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:03,118 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:03,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:03,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:03,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:03,118 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:03,118 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:03,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:03,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:03,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:03,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:03,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:03,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:03,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:03,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:03,121 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:03,121 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:03,121 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:03,121 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:03,121 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:03,121 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:03,123 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:03,123 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:03,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:03,123 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:03,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:03,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:03,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:03,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:03,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:03,123 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:03,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:03,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:03,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:03,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:03,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:03,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:03,125 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:03,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:03,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:03,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:03,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:03,127 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:03,127 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:03,127 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:03,127 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:03,127 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:03,127 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:03,127 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:03,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:03,128 DEBUG [Thread-1741] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:03,127 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:03,128 DEBUG [Thread-1741] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:03,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:03,128 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:03,128 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:03,128 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:03,128 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:03,128 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:55:03,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:03,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:03,170 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:03,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37610 deadline: 1680062163170, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:03,173 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:03,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37570 deadline: 1680062163172, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:03,228 DEBUG [Thread-1741] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:03,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:03,230 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:03,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:03,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:03,231 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:03,231 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:03,231 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:03,231 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:03,231 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:03,232 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:03,233 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:03,233 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:03,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:03,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:03,233 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:03,233 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:03,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:03,234 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:03,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:03,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:03,234 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:03,234 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:03,234 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:03,234 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:03,234 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:03,234 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:03,235 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:03,235 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:03,235 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:55:03,235 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. started... 2023-03-29 03:55:03,235 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. as already flushing 2023-03-29 03:55:03,235 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:55:03,235 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:03,235 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:03,235 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:03,235 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:03,235 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:03,237 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:03,237 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:03,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:03,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:03,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:03,237 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:03,237 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:03,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:03,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:03,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:03,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:03,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:03,239 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:03,239 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16853ff5[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:03,239 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:03,239 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:03,240 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:03,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:03,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:03,240 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:03,240 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:03,240 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:03,240 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:03,240 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:03,243 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:03,243 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:03,243 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:03,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:03,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:03,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:03,243 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:03,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:03,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:03,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:03,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:03,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:03,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:03,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:03,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:03,245 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:03,245 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:03,245 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:03,245 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:03,245 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:03,245 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:03,250 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:03,250 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:03,250 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:03,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:03,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:03,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:03,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:03,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:03,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:03,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:03,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:03,250 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:03,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:03,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:03,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:03,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:03,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:03,252 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:03,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:03,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:03,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:03,254 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:03,254 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:03,254 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:03,254 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:03,254 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:03,254 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:03,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:03,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:03,254 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:03,254 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:03,254 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:03,254 DEBUG [Thread-1741] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:03,254 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:03,254 DEBUG [Thread-1741] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:03,254 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:03,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:03,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:03,255 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:03,264 DEBUG [Thread-1748] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3b4658f8 to 127.0.0.1:52103 2023-03-29 03:55:03,264 DEBUG [Thread-1748] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-29 03:55:03,265 DEBUG [Thread-1750] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5d2b992d to 127.0.0.1:52103 2023-03-29 03:55:03,265 DEBUG [Thread-1750] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-29 03:55:03,265 DEBUG [Thread-1744] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x433f251d to 127.0.0.1:52103 2023-03-29 03:55:03,265 DEBUG [Thread-1744] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-29 03:55:03,266 DEBUG [Thread-1742] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0ea1ca03 to 127.0.0.1:52103 2023-03-29 03:55:03,266 DEBUG [Thread-1742] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-29 03:55:03,267 DEBUG [Thread-1746] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x66ee0bd0 to 127.0.0.1:52103 2023-03-29 03:55:03,267 DEBUG [Thread-1746] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-29 03:55:03,355 DEBUG [Thread-1741] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:03,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:03,373 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:03,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37610 deadline: 1680062163373, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:03,375 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:03,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37570 deadline: 1680062163375, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:03,465 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:03,465 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=520 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/A/b0f0377ed4294e268d6252ec0534f124 2023-03-29 03:55:03,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37572 deadline: 1680062163465, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:03,467 DEBUG [Thread-1733] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8211 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980., hostname=jenkins-hbase4.apache.org,35599,1680062018534, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-29 03:55:03,582 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:03,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 292 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37634 deadline: 1680062163582, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:03,675 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:03,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37610 deadline: 1680062163675, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:03,678 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:03,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37570 deadline: 1680062163678, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:03,876 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=520 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/B/8032ea09078b4e0ca25e4b821260687b 2023-03-29 03:55:04,179 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:04,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37610 deadline: 1680062164179, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:04,180 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:04,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37570 deadline: 1680062164180, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:04,285 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=520 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/C/efc0a57d30f6406abc3449bcecb515d8 2023-03-29 03:55:04,288 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/A/b0f0377ed4294e268d6252ec0534f124 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/b0f0377ed4294e268d6252ec0534f124 2023-03-29 03:55:04,291 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/b0f0377ed4294e268d6252ec0534f124, entries=250, sequenceid=520, filesize=16.7 K 2023-03-29 03:55:04,291 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/B/8032ea09078b4e0ca25e4b821260687b as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/8032ea09078b4e0ca25e4b821260687b 2023-03-29 03:55:04,294 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/8032ea09078b4e0ca25e4b821260687b, entries=150, sequenceid=520, filesize=11.9 K 2023-03-29 03:55:04,295 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/C/efc0a57d30f6406abc3449bcecb515d8 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/efc0a57d30f6406abc3449bcecb515d8 2023-03-29 03:55:04,298 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/efc0a57d30f6406abc3449bcecb515d8, entries=150, sequenceid=520, filesize=11.9 K 2023-03-29 03:55:04,299 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~379.92 KB/389040, currentSize=46.96 KB/48090 for 2ed55fa255954e1310f6b9fd96e28980 in 1243ms, sequenceid=520, compaction requested=true 2023-03-29 03:55:04,299 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2ed55fa255954e1310f6b9fd96e28980: 2023-03-29 03:55:04,299 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:55:04,299 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-29 03:55:04,299 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-29 03:55:04,299 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:55:04,299 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:55:04,300 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 54748 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-29 03:55:04,300 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49868 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-29 03:55:04,300 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1912): 2ed55fa255954e1310f6b9fd96e28980/A is initiating minor compaction (all files) 2023-03-29 03:55:04,300 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1912): 2ed55fa255954e1310f6b9fd96e28980/B is initiating minor compaction (all files) 2023-03-29 03:55:04,300 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2ed55fa255954e1310f6b9fd96e28980/A in TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:55:04,300 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2ed55fa255954e1310f6b9fd96e28980/B in TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:55:04,300 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/ec74989f9707496ca776d9daaa193619, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/b410dc1cfd2043d0abbe715abeb5437d, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/3555d3174aa8401abfb4545673c78212, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/b0f0377ed4294e268d6252ec0534f124] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp, totalSize=53.5 K 2023-03-29 03:55:04,300 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/d3631fed20f1402084bbe0aba71373ff, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/7cac2373fedb4932a4ca882e0779a51b, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/5f12434a444547f48363f426f5376f51, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/8032ea09078b4e0ca25e4b821260687b] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp, totalSize=48.7 K 2023-03-29 03:55:04,301 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting ec74989f9707496ca776d9daaa193619, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=463, earliestPutTs=1680062100010 2023-03-29 03:55:04,301 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting d3631fed20f1402084bbe0aba71373ff, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=463, earliestPutTs=1680062100010 2023-03-29 03:55:04,301 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting b410dc1cfd2043d0abbe715abeb5437d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=484, earliestPutTs=1680062100671 2023-03-29 03:55:04,301 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 7cac2373fedb4932a4ca882e0779a51b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=484, earliestPutTs=1680062100671 2023-03-29 03:55:04,301 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 3555d3174aa8401abfb4545673c78212, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=498, earliestPutTs=1680062101309 2023-03-29 03:55:04,301 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 5f12434a444547f48363f426f5376f51, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=498, earliestPutTs=1680062101309 2023-03-29 03:55:04,302 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting b0f0377ed4294e268d6252ec0534f124, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=520, earliestPutTs=1680062102456 2023-03-29 03:55:04,302 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 8032ea09078b4e0ca25e4b821260687b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=520, earliestPutTs=1680062102456 2023-03-29 03:55:04,308 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2ed55fa255954e1310f6b9fd96e28980#A#compaction#331 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:55:04,308 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] throttle.PressureAwareThroughputController(145): 2ed55fa255954e1310f6b9fd96e28980#B#compaction#332 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:55:04,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on 2ed55fa255954e1310f6b9fd96e28980 2023-03-29 03:55:04,588 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2ed55fa255954e1310f6b9fd96e28980 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-29 03:55:04,588 DEBUG [Thread-1737] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6919b2a0 to 127.0.0.1:52103 2023-03-29 03:55:04,588 DEBUG [Thread-1737] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-29 03:55:04,589 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ed55fa255954e1310f6b9fd96e28980, store=A 2023-03-29 03:55:04,589 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:55:04,589 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ed55fa255954e1310f6b9fd96e28980, store=B 2023-03-29 03:55:04,589 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:55:04,589 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ed55fa255954e1310f6b9fd96e28980, store=C 2023-03-29 03:55:04,589 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:55:04,716 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/A/17c66a19bda04ed1ac271728c80b7fae as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/17c66a19bda04ed1ac271728c80b7fae 2023-03-29 03:55:04,716 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/B/64444e0e34b940148dc6c573e462227c as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/64444e0e34b940148dc6c573e462227c 2023-03-29 03:55:04,720 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 2ed55fa255954e1310f6b9fd96e28980/A of 2ed55fa255954e1310f6b9fd96e28980 into 17c66a19bda04ed1ac271728c80b7fae(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:55:04,720 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2ed55fa255954e1310f6b9fd96e28980: 2023-03-29 03:55:04,720 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980., storeName=2ed55fa255954e1310f6b9fd96e28980/A, priority=12, startTime=1680062104299; duration=0sec 2023-03-29 03:55:04,720 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 2ed55fa255954e1310f6b9fd96e28980/B of 2ed55fa255954e1310f6b9fd96e28980 into 64444e0e34b940148dc6c573e462227c(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:55:04,720 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2ed55fa255954e1310f6b9fd96e28980: 2023-03-29 03:55:04,720 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:55:04,720 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980., storeName=2ed55fa255954e1310f6b9fd96e28980/B, priority=12, startTime=1680062104299; duration=0sec 2023-03-29 03:55:04,720 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-29 03:55:04,720 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:55:04,721 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49868 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-29 03:55:04,721 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1912): 2ed55fa255954e1310f6b9fd96e28980/C is initiating minor compaction (all files) 2023-03-29 03:55:04,721 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2ed55fa255954e1310f6b9fd96e28980/C in TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:55:04,721 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/f94c17bd8cf24fbf82f3a8e9486bac34, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/fb284284b79f446d8f9c1da423f971b7, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/bc662949ab5e47eb80fc5b241600b255, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/efc0a57d30f6406abc3449bcecb515d8] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp, totalSize=48.7 K 2023-03-29 03:55:04,721 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting f94c17bd8cf24fbf82f3a8e9486bac34, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=463, earliestPutTs=1680062100010 2023-03-29 03:55:04,722 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting fb284284b79f446d8f9c1da423f971b7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=484, earliestPutTs=1680062100671 2023-03-29 03:55:04,722 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting bc662949ab5e47eb80fc5b241600b255, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=498, earliestPutTs=1680062101309 2023-03-29 03:55:04,722 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting efc0a57d30f6406abc3449bcecb515d8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=520, earliestPutTs=1680062102456 2023-03-29 03:55:04,728 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2ed55fa255954e1310f6b9fd96e28980#C#compaction#334 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:55:04,996 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=531 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/A/f8fa283fc48c4201ab3c412f26948b5e 2023-03-29 03:55:05,135 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/C/13674d00ec624e3a9ec5c622ce923332 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/13674d00ec624e3a9ec5c622ce923332 2023-03-29 03:55:05,139 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 2ed55fa255954e1310f6b9fd96e28980/C of 2ed55fa255954e1310f6b9fd96e28980 into 13674d00ec624e3a9ec5c622ce923332(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:55:05,139 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2ed55fa255954e1310f6b9fd96e28980: 2023-03-29 03:55:05,139 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980., storeName=2ed55fa255954e1310f6b9fd96e28980/C, priority=12, startTime=1680062104299; duration=0sec 2023-03-29 03:55:05,139 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:55:05,187 DEBUG [Thread-1735] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3cf5bdb7 to 127.0.0.1:52103 2023-03-29 03:55:05,187 DEBUG [Thread-1735] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-29 03:55:05,187 DEBUG [Thread-1731] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5c124820 to 127.0.0.1:52103 2023-03-29 03:55:05,188 DEBUG [Thread-1731] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-29 03:55:05,404 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=531 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/B/c97ef1f16a8e4557a00dff1f2ce672b1 2023-03-29 03:55:05,812 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=531 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/C/7720f118489c4c6c89ff7f6aa7b118d4 2023-03-29 03:55:05,816 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/A/f8fa283fc48c4201ab3c412f26948b5e as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/f8fa283fc48c4201ab3c412f26948b5e 2023-03-29 03:55:05,818 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/f8fa283fc48c4201ab3c412f26948b5e, entries=150, sequenceid=531, filesize=11.9 K 2023-03-29 03:55:05,819 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/B/c97ef1f16a8e4557a00dff1f2ce672b1 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/c97ef1f16a8e4557a00dff1f2ce672b1 2023-03-29 03:55:05,822 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/c97ef1f16a8e4557a00dff1f2ce672b1, entries=150, sequenceid=531, filesize=11.9 K 2023-03-29 03:55:05,823 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/C/7720f118489c4c6c89ff7f6aa7b118d4 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/7720f118489c4c6c89ff7f6aa7b118d4 2023-03-29 03:55:05,826 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/7720f118489c4c6c89ff7f6aa7b118d4, entries=150, sequenceid=531, filesize=11.9 K 2023-03-29 03:55:05,826 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=13.42 KB/13740 for 2ed55fa255954e1310f6b9fd96e28980 in 1238ms, sequenceid=531, compaction requested=false 2023-03-29 03:55:05,826 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2ed55fa255954e1310f6b9fd96e28980: 2023-03-29 03:55:09,764 DEBUG [Thread-1739] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2ce5573e to 127.0.0.1:52103 2023-03-29 03:55:09,764 DEBUG [Thread-1739] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-29 03:55:13,564 DEBUG [Thread-1733] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3157d4df to 127.0.0.1:52103 2023-03-29 03:55:13,564 DEBUG [Thread-1733] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-29 03:55:13,564 INFO [Listener at localhost/35027] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-03-29 03:55:13,564 INFO [Listener at localhost/35027] hbase.AcidGuaranteesTestTool(394): wrote 97 2023-03-29 03:55:13,564 INFO [Listener at localhost/35027] hbase.AcidGuaranteesTestTool(394): wrote 59 2023-03-29 03:55:13,564 INFO [Listener at localhost/35027] hbase.AcidGuaranteesTestTool(394): wrote 100 2023-03-29 03:55:13,564 INFO [Listener at localhost/35027] hbase.AcidGuaranteesTestTool(394): wrote 131 2023-03-29 03:55:13,564 INFO [Listener at localhost/35027] hbase.AcidGuaranteesTestTool(394): wrote 31 2023-03-29 03:55:13,564 INFO [Listener at localhost/35027] hbase.AcidGuaranteesTestTool(396): Readers: 2023-03-29 03:55:13,564 INFO [Listener at localhost/35027] hbase.AcidGuaranteesTestTool(398): read 5671 2023-03-29 03:55:13,564 INFO [Listener at localhost/35027] hbase.AcidGuaranteesTestTool(398): read 5671 2023-03-29 03:55:13,564 INFO [Listener at localhost/35027] hbase.AcidGuaranteesTestTool(398): read 5478 2023-03-29 03:55:13,564 INFO [Listener at localhost/35027] hbase.AcidGuaranteesTestTool(398): read 5676 2023-03-29 03:55:13,564 INFO [Listener at localhost/35027] hbase.AcidGuaranteesTestTool(398): read 5635 2023-03-29 03:55:13,564 INFO [Listener at localhost/35027] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-03-29 03:55:13,565 INFO [Listener at localhost/35027] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-03-29 03:55:13,565 DEBUG [Listener at localhost/35027] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x23c97092 to 127.0.0.1:52103 2023-03-29 03:55:13,565 DEBUG [Listener at localhost/35027] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-29 03:55:13,565 INFO [Listener at localhost/35027] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-03-29 03:55:13,565 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-03-29 03:55:13,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure2.ProcedureExecutor(1029): Stored pid=31, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-03-29 03:55:13,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-03-29 03:55:13,569 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680062113569"}]},"ts":"1680062113569"} 2023-03-29 03:55:13,569 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-03-29 03:55:13,571 INFO [PEWorker-2] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-03-29 03:55:13,572 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=32, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=2ed55fa255954e1310f6b9fd96e28980, UNASSIGN}] 2023-03-29 03:55:13,573 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=32, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=2ed55fa255954e1310f6b9fd96e28980, UNASSIGN 2023-03-29 03:55:13,574 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=2ed55fa255954e1310f6b9fd96e28980, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:13,574 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680062113574"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680062113574"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680062113574"}]},"ts":"1680062113574"} 2023-03-29 03:55:13,575 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=33, ppid=32, state=RUNNABLE; CloseRegionProcedure 2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534}] 2023-03-29 03:55:13,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-03-29 03:55:13,727 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 2ed55fa255954e1310f6b9fd96e28980 2023-03-29 03:55:13,727 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 2ed55fa255954e1310f6b9fd96e28980, disabling compactions & flushes 2023-03-29 03:55:13,727 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:55:13,727 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:55:13,727 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. after waiting 0 ms 2023-03-29 03:55:13,727 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:55:13,727 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 2ed55fa255954e1310f6b9fd96e28980 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-03-29 03:55:13,728 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ed55fa255954e1310f6b9fd96e28980, store=A 2023-03-29 03:55:13,728 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:55:13,728 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ed55fa255954e1310f6b9fd96e28980, store=B 2023-03-29 03:55:13,728 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:55:13,728 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2ed55fa255954e1310f6b9fd96e28980, store=C 2023-03-29 03:55:13,728 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:55:13,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-03-29 03:55:14,134 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=541 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/A/bac72583ed294fe88830b9b7c0e6846e 2023-03-29 03:55:14,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-03-29 03:55:14,543 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=541 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/B/cc438b22c6ca4260b9471aca5f68a469 2023-03-29 03:55:14,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-03-29 03:55:14,951 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=541 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/C/4e1b1006219a4664b6ff9abfbc6ae95f 2023-03-29 03:55:14,954 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/A/bac72583ed294fe88830b9b7c0e6846e as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/bac72583ed294fe88830b9b7c0e6846e 2023-03-29 03:55:14,957 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/bac72583ed294fe88830b9b7c0e6846e, entries=100, sequenceid=541, filesize=9.5 K 2023-03-29 03:55:14,958 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/B/cc438b22c6ca4260b9471aca5f68a469 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/cc438b22c6ca4260b9471aca5f68a469 2023-03-29 03:55:14,961 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/cc438b22c6ca4260b9471aca5f68a469, entries=100, sequenceid=541, filesize=9.5 K 2023-03-29 03:55:14,961 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/.tmp/C/4e1b1006219a4664b6ff9abfbc6ae95f as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/4e1b1006219a4664b6ff9abfbc6ae95f 2023-03-29 03:55:14,964 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/4e1b1006219a4664b6ff9abfbc6ae95f, entries=100, sequenceid=541, filesize=9.5 K 2023-03-29 03:55:14,965 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for 2ed55fa255954e1310f6b9fd96e28980 in 1238ms, sequenceid=541, compaction requested=true 2023-03-29 03:55:14,986 DEBUG [StoreCloser-TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/d5416a3ab8c94b59b205a497e7ccf908, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/62af4785ca5e4f18b6bf0ebc2f1027fb, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/fb37541f10a54c0aa4f04af8de268e0c, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/0f33d61018ff46b48f01ca6764fce7d1, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/b884545047f64c17b7949a2c6b266711, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/83e4364ed4be4bbabeae2258adbf0001, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/a6b5f9126e2c4320ac1f6421a8063c4b, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/39b6b51e018d43d19d6c3de0f0f27f0d, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/92624db90abe44d394a95b0e9b0577be, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/560f1517f0c4405c8668da436fbfeb5a, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/20232a5b156349208b03a6a724b74839, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/bb9be275b7ff43c9ac73f6546f136afa, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/8dd38d24787c4d5e9ce4d68e3ffb11ec, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/9b264a63081c4a338b515f7228a0e07d, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/541188092080413aaaf5a9585a5d57c3, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/5a2f8b2971fd4ceb99d780f2ff43116e, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/bdc9d545c90f4638ad8bb52924252650, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/4d20d8dcbd8a4f0e808fe0b86874bcff, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/96d79fb5dc124d8b93152ccf431fb98c, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/e5a831d240cc4946ae03c60f2b9a9cf9, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/7585ecb9f4564853b69199ca5babf598, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/db0999310d304847bdb4d5cecf3f57f8, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/cf5c7d1773db435ba216bb8382cf9b6a, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/607b40de47d04783a23e000d67520b6f, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/399637a46e644dceb34922f1d2962c6f, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/1f64b87f85074eb5b6abbcbbae1b86eb, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/98e78e88fa334457a10d8d8bafcc31ac, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/932e0c76313543b29e70d1b8be69fff8, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/31dbac4d1e9a4fea893ebbe1edd899dc, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/7fe8a3f221f7463bae715f263264e4f3, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/55b33dcdf6c9410e9c965495e2a87d76, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/76468a509c1048e68ef9763c0b34c84d, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/30195084e4e34157ae842f3c81370e78, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/ec74989f9707496ca776d9daaa193619, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/b410dc1cfd2043d0abbe715abeb5437d, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/3555d3174aa8401abfb4545673c78212, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/b0f0377ed4294e268d6252ec0534f124] to archive 2023-03-29 03:55:14,987 DEBUG [StoreCloser-TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-29 03:55:14,988 DEBUG [StoreCloser-TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/d5416a3ab8c94b59b205a497e7ccf908 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/d5416a3ab8c94b59b205a497e7ccf908 2023-03-29 03:55:14,989 DEBUG [StoreCloser-TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/62af4785ca5e4f18b6bf0ebc2f1027fb to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/62af4785ca5e4f18b6bf0ebc2f1027fb 2023-03-29 03:55:14,990 DEBUG [StoreCloser-TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/fb37541f10a54c0aa4f04af8de268e0c to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/fb37541f10a54c0aa4f04af8de268e0c 2023-03-29 03:55:14,991 DEBUG [StoreCloser-TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/0f33d61018ff46b48f01ca6764fce7d1 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/0f33d61018ff46b48f01ca6764fce7d1 2023-03-29 03:55:14,992 DEBUG [StoreCloser-TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/b884545047f64c17b7949a2c6b266711 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/b884545047f64c17b7949a2c6b266711 2023-03-29 03:55:14,993 DEBUG [StoreCloser-TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/83e4364ed4be4bbabeae2258adbf0001 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/83e4364ed4be4bbabeae2258adbf0001 2023-03-29 03:55:14,993 DEBUG [StoreCloser-TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/a6b5f9126e2c4320ac1f6421a8063c4b to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/a6b5f9126e2c4320ac1f6421a8063c4b 2023-03-29 03:55:14,994 DEBUG [StoreCloser-TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/39b6b51e018d43d19d6c3de0f0f27f0d to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/39b6b51e018d43d19d6c3de0f0f27f0d 2023-03-29 03:55:14,995 DEBUG [StoreCloser-TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/92624db90abe44d394a95b0e9b0577be to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/92624db90abe44d394a95b0e9b0577be 2023-03-29 03:55:14,996 DEBUG [StoreCloser-TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/560f1517f0c4405c8668da436fbfeb5a to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/560f1517f0c4405c8668da436fbfeb5a 2023-03-29 03:55:14,997 DEBUG [StoreCloser-TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/20232a5b156349208b03a6a724b74839 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/20232a5b156349208b03a6a724b74839 2023-03-29 03:55:14,998 DEBUG [StoreCloser-TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/bb9be275b7ff43c9ac73f6546f136afa to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/bb9be275b7ff43c9ac73f6546f136afa 2023-03-29 03:55:14,999 DEBUG [StoreCloser-TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/8dd38d24787c4d5e9ce4d68e3ffb11ec to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/8dd38d24787c4d5e9ce4d68e3ffb11ec 2023-03-29 03:55:15,000 DEBUG [StoreCloser-TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/9b264a63081c4a338b515f7228a0e07d to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/9b264a63081c4a338b515f7228a0e07d 2023-03-29 03:55:15,001 DEBUG [StoreCloser-TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/541188092080413aaaf5a9585a5d57c3 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/541188092080413aaaf5a9585a5d57c3 2023-03-29 03:55:15,001 DEBUG [StoreCloser-TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/5a2f8b2971fd4ceb99d780f2ff43116e to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/5a2f8b2971fd4ceb99d780f2ff43116e 2023-03-29 03:55:15,002 DEBUG [StoreCloser-TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/bdc9d545c90f4638ad8bb52924252650 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/bdc9d545c90f4638ad8bb52924252650 2023-03-29 03:55:15,003 DEBUG [StoreCloser-TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/4d20d8dcbd8a4f0e808fe0b86874bcff to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/4d20d8dcbd8a4f0e808fe0b86874bcff 2023-03-29 03:55:15,004 DEBUG [StoreCloser-TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/96d79fb5dc124d8b93152ccf431fb98c to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/96d79fb5dc124d8b93152ccf431fb98c 2023-03-29 03:55:15,005 DEBUG [StoreCloser-TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/e5a831d240cc4946ae03c60f2b9a9cf9 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/e5a831d240cc4946ae03c60f2b9a9cf9 2023-03-29 03:55:15,006 DEBUG [StoreCloser-TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/7585ecb9f4564853b69199ca5babf598 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/7585ecb9f4564853b69199ca5babf598 2023-03-29 03:55:15,007 DEBUG [StoreCloser-TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/db0999310d304847bdb4d5cecf3f57f8 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/db0999310d304847bdb4d5cecf3f57f8 2023-03-29 03:55:15,008 DEBUG [StoreCloser-TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/cf5c7d1773db435ba216bb8382cf9b6a to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/cf5c7d1773db435ba216bb8382cf9b6a 2023-03-29 03:55:15,009 DEBUG [StoreCloser-TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/607b40de47d04783a23e000d67520b6f to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/607b40de47d04783a23e000d67520b6f 2023-03-29 03:55:15,009 DEBUG [StoreCloser-TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/399637a46e644dceb34922f1d2962c6f to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/399637a46e644dceb34922f1d2962c6f 2023-03-29 03:55:15,010 DEBUG [StoreCloser-TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/1f64b87f85074eb5b6abbcbbae1b86eb to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/1f64b87f85074eb5b6abbcbbae1b86eb 2023-03-29 03:55:15,011 DEBUG [StoreCloser-TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/98e78e88fa334457a10d8d8bafcc31ac to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/98e78e88fa334457a10d8d8bafcc31ac 2023-03-29 03:55:15,012 DEBUG [StoreCloser-TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/932e0c76313543b29e70d1b8be69fff8 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/932e0c76313543b29e70d1b8be69fff8 2023-03-29 03:55:15,013 DEBUG [StoreCloser-TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/31dbac4d1e9a4fea893ebbe1edd899dc to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/31dbac4d1e9a4fea893ebbe1edd899dc 2023-03-29 03:55:15,014 DEBUG [StoreCloser-TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/7fe8a3f221f7463bae715f263264e4f3 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/7fe8a3f221f7463bae715f263264e4f3 2023-03-29 03:55:15,014 DEBUG [StoreCloser-TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/55b33dcdf6c9410e9c965495e2a87d76 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/55b33dcdf6c9410e9c965495e2a87d76 2023-03-29 03:55:15,015 DEBUG [StoreCloser-TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/76468a509c1048e68ef9763c0b34c84d to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/76468a509c1048e68ef9763c0b34c84d 2023-03-29 03:55:15,016 DEBUG [StoreCloser-TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/30195084e4e34157ae842f3c81370e78 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/30195084e4e34157ae842f3c81370e78 2023-03-29 03:55:15,017 DEBUG [StoreCloser-TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/ec74989f9707496ca776d9daaa193619 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/ec74989f9707496ca776d9daaa193619 2023-03-29 03:55:15,018 DEBUG [StoreCloser-TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/b410dc1cfd2043d0abbe715abeb5437d to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/b410dc1cfd2043d0abbe715abeb5437d 2023-03-29 03:55:15,018 DEBUG [StoreCloser-TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/3555d3174aa8401abfb4545673c78212 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/3555d3174aa8401abfb4545673c78212 2023-03-29 03:55:15,019 DEBUG [StoreCloser-TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/b0f0377ed4294e268d6252ec0534f124 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/b0f0377ed4294e268d6252ec0534f124 2023-03-29 03:55:15,022 DEBUG [StoreCloser-TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/35a0fd77f91045d795cab8be37f29b68, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/08fbb1b29f924230963c70f920d33799, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/2e93d88526c5484fbd124c2e5ffd8ed0, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/a5bb8db8363043c1907c8806bfff2078, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/37cbfd98fc4549a0b43ef46399fe115e, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/f57b2b6aed914d78b8b3510ab1b4cba6, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/02c751b5b36240969d39909a28ec3412, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/b7e543181c5f490e83467ca868a07b8e, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/f1f8d58b22c04f5ea07843af4b74bedf, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/71ca5d31c1244abea369822ff32bd33b, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/b608f0638f974340b1bc10205e9b4174, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/7daa29719f94484589b9f3faa4d7640c, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/33e41c49ed7c45a98d149b787652e796, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/c46ac230c2cb4b12b740ee7b37621a55, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/054a5dcdbed044a8b6dedb4201f32c08, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/9e0a1699f9294c21a2c3bc9346973d84, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/174ddd6b9710415480e0e1c0a2a0f76a, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/f11c9143d9b541f5b7526fe38cb30d41, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/23c9506a31bb4fd28af54f0add74f389, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/e40baabca095450f9e91c9ee6137a521, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/4be51dbc1b2f4086bc18a40fcb7aa564, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/e27e87bf31de4adfb06585b29ffbbcda, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/b31a4196edd24c88aed1c8646aafd41c, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/c209d746f09f47df935bae6ac4905037, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/0ffd1d8e06b940e2b5bba5f3a72e2958, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/1b6c5a6406044890b8993cc36b060672, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/7fd9d678aa4a4150afd006f684754c24, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/b23952d5900849b693e69ff429be80c3, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/5c5f03cf6e2a49a1a95cec2c04ccee37, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/976f7d2e5d8e45789a99f5d11c4b7ac7, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/2173a9cdc57b46699006a7fdb8bef655, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/9582ecfeab924bfbb38b5b4e0d40aa3e, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/d3631fed20f1402084bbe0aba71373ff, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/498b866acfdd45689752d73d71e8b36b, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/7cac2373fedb4932a4ca882e0779a51b, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/5f12434a444547f48363f426f5376f51, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/8032ea09078b4e0ca25e4b821260687b] to archive 2023-03-29 03:55:15,023 DEBUG [StoreCloser-TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-29 03:55:15,024 DEBUG [StoreCloser-TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/35a0fd77f91045d795cab8be37f29b68 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/35a0fd77f91045d795cab8be37f29b68 2023-03-29 03:55:15,025 DEBUG [StoreCloser-TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/08fbb1b29f924230963c70f920d33799 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/08fbb1b29f924230963c70f920d33799 2023-03-29 03:55:15,036 DEBUG [StoreCloser-TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/2e93d88526c5484fbd124c2e5ffd8ed0 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/2e93d88526c5484fbd124c2e5ffd8ed0 2023-03-29 03:55:15,037 DEBUG [StoreCloser-TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/a5bb8db8363043c1907c8806bfff2078 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/a5bb8db8363043c1907c8806bfff2078 2023-03-29 03:55:15,038 DEBUG [StoreCloser-TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/37cbfd98fc4549a0b43ef46399fe115e to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/37cbfd98fc4549a0b43ef46399fe115e 2023-03-29 03:55:15,038 DEBUG [StoreCloser-TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/f57b2b6aed914d78b8b3510ab1b4cba6 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/f57b2b6aed914d78b8b3510ab1b4cba6 2023-03-29 03:55:15,039 DEBUG [StoreCloser-TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/02c751b5b36240969d39909a28ec3412 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/02c751b5b36240969d39909a28ec3412 2023-03-29 03:55:15,040 DEBUG [StoreCloser-TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/b7e543181c5f490e83467ca868a07b8e to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/b7e543181c5f490e83467ca868a07b8e 2023-03-29 03:55:15,041 DEBUG [StoreCloser-TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/f1f8d58b22c04f5ea07843af4b74bedf to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/f1f8d58b22c04f5ea07843af4b74bedf 2023-03-29 03:55:15,042 DEBUG [StoreCloser-TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/71ca5d31c1244abea369822ff32bd33b to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/71ca5d31c1244abea369822ff32bd33b 2023-03-29 03:55:15,043 DEBUG [StoreCloser-TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/b608f0638f974340b1bc10205e9b4174 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/b608f0638f974340b1bc10205e9b4174 2023-03-29 03:55:15,044 DEBUG [StoreCloser-TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/7daa29719f94484589b9f3faa4d7640c to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/7daa29719f94484589b9f3faa4d7640c 2023-03-29 03:55:15,045 DEBUG [StoreCloser-TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/33e41c49ed7c45a98d149b787652e796 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/33e41c49ed7c45a98d149b787652e796 2023-03-29 03:55:15,045 DEBUG [StoreCloser-TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/c46ac230c2cb4b12b740ee7b37621a55 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/c46ac230c2cb4b12b740ee7b37621a55 2023-03-29 03:55:15,046 DEBUG [StoreCloser-TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/054a5dcdbed044a8b6dedb4201f32c08 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/054a5dcdbed044a8b6dedb4201f32c08 2023-03-29 03:55:15,047 DEBUG [StoreCloser-TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/9e0a1699f9294c21a2c3bc9346973d84 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/9e0a1699f9294c21a2c3bc9346973d84 2023-03-29 03:55:15,048 DEBUG [StoreCloser-TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/174ddd6b9710415480e0e1c0a2a0f76a to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/174ddd6b9710415480e0e1c0a2a0f76a 2023-03-29 03:55:15,049 DEBUG [StoreCloser-TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/f11c9143d9b541f5b7526fe38cb30d41 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/f11c9143d9b541f5b7526fe38cb30d41 2023-03-29 03:55:15,050 DEBUG [StoreCloser-TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/23c9506a31bb4fd28af54f0add74f389 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/23c9506a31bb4fd28af54f0add74f389 2023-03-29 03:55:15,051 DEBUG [StoreCloser-TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/e40baabca095450f9e91c9ee6137a521 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/e40baabca095450f9e91c9ee6137a521 2023-03-29 03:55:15,051 DEBUG [StoreCloser-TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/4be51dbc1b2f4086bc18a40fcb7aa564 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/4be51dbc1b2f4086bc18a40fcb7aa564 2023-03-29 03:55:15,052 DEBUG [StoreCloser-TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/e27e87bf31de4adfb06585b29ffbbcda to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/e27e87bf31de4adfb06585b29ffbbcda 2023-03-29 03:55:15,053 DEBUG [StoreCloser-TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/b31a4196edd24c88aed1c8646aafd41c to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/b31a4196edd24c88aed1c8646aafd41c 2023-03-29 03:55:15,054 DEBUG [StoreCloser-TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/c209d746f09f47df935bae6ac4905037 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/c209d746f09f47df935bae6ac4905037 2023-03-29 03:55:15,055 DEBUG [StoreCloser-TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/0ffd1d8e06b940e2b5bba5f3a72e2958 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/0ffd1d8e06b940e2b5bba5f3a72e2958 2023-03-29 03:55:15,055 DEBUG [StoreCloser-TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/1b6c5a6406044890b8993cc36b060672 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/1b6c5a6406044890b8993cc36b060672 2023-03-29 03:55:15,056 DEBUG [StoreCloser-TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/7fd9d678aa4a4150afd006f684754c24 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/7fd9d678aa4a4150afd006f684754c24 2023-03-29 03:55:15,057 DEBUG [StoreCloser-TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/b23952d5900849b693e69ff429be80c3 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/b23952d5900849b693e69ff429be80c3 2023-03-29 03:55:15,058 DEBUG [StoreCloser-TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/5c5f03cf6e2a49a1a95cec2c04ccee37 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/5c5f03cf6e2a49a1a95cec2c04ccee37 2023-03-29 03:55:15,059 DEBUG [StoreCloser-TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/976f7d2e5d8e45789a99f5d11c4b7ac7 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/976f7d2e5d8e45789a99f5d11c4b7ac7 2023-03-29 03:55:15,059 DEBUG [StoreCloser-TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/2173a9cdc57b46699006a7fdb8bef655 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/2173a9cdc57b46699006a7fdb8bef655 2023-03-29 03:55:15,060 DEBUG [StoreCloser-TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/9582ecfeab924bfbb38b5b4e0d40aa3e to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/9582ecfeab924bfbb38b5b4e0d40aa3e 2023-03-29 03:55:15,061 DEBUG [StoreCloser-TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/d3631fed20f1402084bbe0aba71373ff to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/d3631fed20f1402084bbe0aba71373ff 2023-03-29 03:55:15,062 DEBUG [StoreCloser-TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/498b866acfdd45689752d73d71e8b36b to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/498b866acfdd45689752d73d71e8b36b 2023-03-29 03:55:15,063 DEBUG [StoreCloser-TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/7cac2373fedb4932a4ca882e0779a51b to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/7cac2373fedb4932a4ca882e0779a51b 2023-03-29 03:55:15,064 DEBUG [StoreCloser-TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/5f12434a444547f48363f426f5376f51 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/5f12434a444547f48363f426f5376f51 2023-03-29 03:55:15,064 DEBUG [StoreCloser-TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/8032ea09078b4e0ca25e4b821260687b to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/8032ea09078b4e0ca25e4b821260687b 2023-03-29 03:55:15,068 DEBUG [StoreCloser-TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/5a5183a24464403daf8eee49ef0652d3, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/6e831407f49d477b94aa1daa8a1564ba, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/91ee23eb360842a29ac70105fa934f9f, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/d629ddeaf72640d8bcc395035d9326d7, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/ea1581cf7b08437dbfd19c053685f48d, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/6cab4833de0746db9dab5db3d6dfc5c7, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/a0723fc844e6448dbec6821a9d0a2493, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/77de40d9d4b2449ab7e381b4182db937, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/5b7c7b9c1c814a2283cb69858846a885, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/2ac363cfe65a4ad3b584c6745815d5e8, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/3485dbc33ab94d5e9268ad6c63314712, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/9762d43c64e540c794ea2ebbbf2c009e, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/d2c5711d1d274e51b45d4ad197801f4f, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/92128bded558423881b376eacfd1c90e, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/abe79ae3798b48d9a58fbc0c99178013, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/da7a8b072069472e9a74756697f88520, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/0f5f678ea4964ffe857b2b58b01d2c8e, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/977142afe896430c8962ea3bdf5f054b, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/a33da568269e4831816da56edc5f8cc6, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/20059f61c4214d0891e5653b6d4dbd17, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/afce92aa5daa4ef3a32515a231e15986, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/ff30a175e2014f7287d14a2e90f29d98, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/bcf82d1371cf4d788d18104ce5e497ba, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/0a840e43c8fa48c9b41d949254f71fc5, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/75e5c22bbee4423a9a0c04eade018515, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/78dd16f5f1c441d08f5100f41a11211e, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/eb29b1a2903b4523baab76616d7b596c, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/7f5458004d224d53b183f0cca1bdf866, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/57dbfc8899bd4b4197ff042894dac7f7, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/00dec3b578e14a1db005e73ec33c4d14, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/867eb19dc0d34e5a8c33f0d2cd4e637f, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/186760cc625d4bffb865ec589aea5142, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/f94c17bd8cf24fbf82f3a8e9486bac34, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/cb956507fe85419e868d7c6e5b73d4a2, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/fb284284b79f446d8f9c1da423f971b7, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/bc662949ab5e47eb80fc5b241600b255, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/efc0a57d30f6406abc3449bcecb515d8] to archive 2023-03-29 03:55:15,069 DEBUG [StoreCloser-TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-29 03:55:15,070 DEBUG [StoreCloser-TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/5a5183a24464403daf8eee49ef0652d3 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/5a5183a24464403daf8eee49ef0652d3 2023-03-29 03:55:15,071 DEBUG [StoreCloser-TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/6e831407f49d477b94aa1daa8a1564ba to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/6e831407f49d477b94aa1daa8a1564ba 2023-03-29 03:55:15,072 DEBUG [StoreCloser-TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/91ee23eb360842a29ac70105fa934f9f to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/91ee23eb360842a29ac70105fa934f9f 2023-03-29 03:55:15,073 DEBUG [StoreCloser-TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/d629ddeaf72640d8bcc395035d9326d7 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/d629ddeaf72640d8bcc395035d9326d7 2023-03-29 03:55:15,074 DEBUG [StoreCloser-TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/ea1581cf7b08437dbfd19c053685f48d to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/ea1581cf7b08437dbfd19c053685f48d 2023-03-29 03:55:15,075 DEBUG [StoreCloser-TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/6cab4833de0746db9dab5db3d6dfc5c7 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/6cab4833de0746db9dab5db3d6dfc5c7 2023-03-29 03:55:15,076 DEBUG [StoreCloser-TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/a0723fc844e6448dbec6821a9d0a2493 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/a0723fc844e6448dbec6821a9d0a2493 2023-03-29 03:55:15,077 DEBUG [StoreCloser-TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/77de40d9d4b2449ab7e381b4182db937 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/77de40d9d4b2449ab7e381b4182db937 2023-03-29 03:55:15,078 DEBUG [StoreCloser-TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/5b7c7b9c1c814a2283cb69858846a885 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/5b7c7b9c1c814a2283cb69858846a885 2023-03-29 03:55:15,079 DEBUG [StoreCloser-TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/2ac363cfe65a4ad3b584c6745815d5e8 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/2ac363cfe65a4ad3b584c6745815d5e8 2023-03-29 03:55:15,080 DEBUG [StoreCloser-TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/3485dbc33ab94d5e9268ad6c63314712 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/3485dbc33ab94d5e9268ad6c63314712 2023-03-29 03:55:15,081 DEBUG [StoreCloser-TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/9762d43c64e540c794ea2ebbbf2c009e to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/9762d43c64e540c794ea2ebbbf2c009e 2023-03-29 03:55:15,082 DEBUG [StoreCloser-TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/d2c5711d1d274e51b45d4ad197801f4f to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/d2c5711d1d274e51b45d4ad197801f4f 2023-03-29 03:55:15,083 DEBUG [StoreCloser-TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/92128bded558423881b376eacfd1c90e to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/92128bded558423881b376eacfd1c90e 2023-03-29 03:55:15,083 DEBUG [StoreCloser-TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/abe79ae3798b48d9a58fbc0c99178013 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/abe79ae3798b48d9a58fbc0c99178013 2023-03-29 03:55:15,084 DEBUG [StoreCloser-TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/da7a8b072069472e9a74756697f88520 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/da7a8b072069472e9a74756697f88520 2023-03-29 03:55:15,085 DEBUG [StoreCloser-TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/0f5f678ea4964ffe857b2b58b01d2c8e to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/0f5f678ea4964ffe857b2b58b01d2c8e 2023-03-29 03:55:15,086 DEBUG [StoreCloser-TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/977142afe896430c8962ea3bdf5f054b to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/977142afe896430c8962ea3bdf5f054b 2023-03-29 03:55:15,087 DEBUG [StoreCloser-TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/a33da568269e4831816da56edc5f8cc6 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/a33da568269e4831816da56edc5f8cc6 2023-03-29 03:55:15,088 DEBUG [StoreCloser-TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/20059f61c4214d0891e5653b6d4dbd17 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/20059f61c4214d0891e5653b6d4dbd17 2023-03-29 03:55:15,088 DEBUG [StoreCloser-TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/afce92aa5daa4ef3a32515a231e15986 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/afce92aa5daa4ef3a32515a231e15986 2023-03-29 03:55:15,089 DEBUG [StoreCloser-TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/ff30a175e2014f7287d14a2e90f29d98 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/ff30a175e2014f7287d14a2e90f29d98 2023-03-29 03:55:15,090 DEBUG [StoreCloser-TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/bcf82d1371cf4d788d18104ce5e497ba to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/bcf82d1371cf4d788d18104ce5e497ba 2023-03-29 03:55:15,091 DEBUG [StoreCloser-TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/0a840e43c8fa48c9b41d949254f71fc5 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/0a840e43c8fa48c9b41d949254f71fc5 2023-03-29 03:55:15,091 DEBUG [StoreCloser-TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/75e5c22bbee4423a9a0c04eade018515 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/75e5c22bbee4423a9a0c04eade018515 2023-03-29 03:55:15,092 DEBUG [StoreCloser-TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/78dd16f5f1c441d08f5100f41a11211e to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/78dd16f5f1c441d08f5100f41a11211e 2023-03-29 03:55:15,093 DEBUG [StoreCloser-TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/eb29b1a2903b4523baab76616d7b596c to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/eb29b1a2903b4523baab76616d7b596c 2023-03-29 03:55:15,094 DEBUG [StoreCloser-TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/7f5458004d224d53b183f0cca1bdf866 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/7f5458004d224d53b183f0cca1bdf866 2023-03-29 03:55:15,095 DEBUG [StoreCloser-TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/57dbfc8899bd4b4197ff042894dac7f7 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/57dbfc8899bd4b4197ff042894dac7f7 2023-03-29 03:55:15,095 DEBUG [StoreCloser-TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/00dec3b578e14a1db005e73ec33c4d14 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/00dec3b578e14a1db005e73ec33c4d14 2023-03-29 03:55:15,096 DEBUG [StoreCloser-TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/867eb19dc0d34e5a8c33f0d2cd4e637f to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/867eb19dc0d34e5a8c33f0d2cd4e637f 2023-03-29 03:55:15,097 DEBUG [StoreCloser-TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/186760cc625d4bffb865ec589aea5142 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/186760cc625d4bffb865ec589aea5142 2023-03-29 03:55:15,097 DEBUG [StoreCloser-TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/f94c17bd8cf24fbf82f3a8e9486bac34 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/f94c17bd8cf24fbf82f3a8e9486bac34 2023-03-29 03:55:15,098 DEBUG [StoreCloser-TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/cb956507fe85419e868d7c6e5b73d4a2 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/cb956507fe85419e868d7c6e5b73d4a2 2023-03-29 03:55:15,099 DEBUG [StoreCloser-TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/fb284284b79f446d8f9c1da423f971b7 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/fb284284b79f446d8f9c1da423f971b7 2023-03-29 03:55:15,100 DEBUG [StoreCloser-TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/bc662949ab5e47eb80fc5b241600b255 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/bc662949ab5e47eb80fc5b241600b255 2023-03-29 03:55:15,101 DEBUG [StoreCloser-TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/efc0a57d30f6406abc3449bcecb515d8 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/efc0a57d30f6406abc3449bcecb515d8 2023-03-29 03:55:15,104 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/recovered.edits/544.seqid, newMaxSeqId=544, maxSeqId=1 2023-03-29 03:55:15,106 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980. 2023-03-29 03:55:15,106 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 2ed55fa255954e1310f6b9fd96e28980: 2023-03-29 03:55:15,107 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 2ed55fa255954e1310f6b9fd96e28980 2023-03-29 03:55:15,107 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=2ed55fa255954e1310f6b9fd96e28980, regionState=CLOSED 2023-03-29 03:55:15,107 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680062115107"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680062115107"}]},"ts":"1680062115107"} 2023-03-29 03:55:15,109 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=33, resume processing ppid=32 2023-03-29 03:55:15,110 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=33, ppid=32, state=SUCCESS; CloseRegionProcedure 2ed55fa255954e1310f6b9fd96e28980, server=jenkins-hbase4.apache.org,35599,1680062018534 in 1.5330 sec 2023-03-29 03:55:15,111 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=32, resume processing ppid=31 2023-03-29 03:55:15,111 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=32, ppid=31, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=2ed55fa255954e1310f6b9fd96e28980, UNASSIGN in 1.5370 sec 2023-03-29 03:55:15,111 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680062115111"}]},"ts":"1680062115111"} 2023-03-29 03:55:15,112 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-03-29 03:55:15,113 INFO [PEWorker-2] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-03-29 03:55:15,115 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=31, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.5480 sec 2023-03-29 03:55:15,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-03-29 03:55:15,672 INFO [Listener at localhost/35027] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 31 completed 2023-03-29 03:55:15,673 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-03-29 03:55:15,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure2.ProcedureExecutor(1029): Stored pid=34, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-03-29 03:55:15,675 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=34, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-29 03:55:15,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=34 2023-03-29 03:55:15,675 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=34, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-29 03:55:15,679 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/.tmp/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980 2023-03-29 03:55:15,681 DEBUG [HFileArchiver-7] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/.tmp/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A, FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/.tmp/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B, FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/.tmp/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C, FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/.tmp/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/recovered.edits] 2023-03-29 03:55:15,683 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/.tmp/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/17c66a19bda04ed1ac271728c80b7fae to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/17c66a19bda04ed1ac271728c80b7fae 2023-03-29 03:55:15,684 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/.tmp/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/bac72583ed294fe88830b9b7c0e6846e to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/bac72583ed294fe88830b9b7c0e6846e 2023-03-29 03:55:15,685 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/.tmp/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/f8fa283fc48c4201ab3c412f26948b5e to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/A/f8fa283fc48c4201ab3c412f26948b5e 2023-03-29 03:55:15,687 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/.tmp/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/64444e0e34b940148dc6c573e462227c to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/64444e0e34b940148dc6c573e462227c 2023-03-29 03:55:15,688 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/.tmp/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/c97ef1f16a8e4557a00dff1f2ce672b1 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/c97ef1f16a8e4557a00dff1f2ce672b1 2023-03-29 03:55:15,689 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/.tmp/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/cc438b22c6ca4260b9471aca5f68a469 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/B/cc438b22c6ca4260b9471aca5f68a469 2023-03-29 03:55:15,691 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/.tmp/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/13674d00ec624e3a9ec5c622ce923332 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/13674d00ec624e3a9ec5c622ce923332 2023-03-29 03:55:15,692 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/.tmp/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/4e1b1006219a4664b6ff9abfbc6ae95f to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/4e1b1006219a4664b6ff9abfbc6ae95f 2023-03-29 03:55:15,693 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/.tmp/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/7720f118489c4c6c89ff7f6aa7b118d4 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/C/7720f118489c4c6c89ff7f6aa7b118d4 2023-03-29 03:55:15,695 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/.tmp/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/recovered.edits/544.seqid to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980/recovered.edits/544.seqid 2023-03-29 03:55:15,695 DEBUG [HFileArchiver-7] backup.HFileArchiver(596): Deleted hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/.tmp/data/default/TestAcidGuarantees/2ed55fa255954e1310f6b9fd96e28980 2023-03-29 03:55:15,695 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-29 03:55:15,697 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=34, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-29 03:55:15,698 WARN [PEWorker-5] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-03-29 03:55:15,700 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-03-29 03:55:15,701 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=34, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-29 03:55:15,701 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-03-29 03:55:15,701 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1680062115701"}]},"ts":"9223372036854775807"} 2023-03-29 03:55:15,702 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-03-29 03:55:15,702 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 2ed55fa255954e1310f6b9fd96e28980, NAME => 'TestAcidGuarantees,,1680062081098.2ed55fa255954e1310f6b9fd96e28980.', STARTKEY => '', ENDKEY => ''}] 2023-03-29 03:55:15,702 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-03-29 03:55:15,702 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1680062115702"}]},"ts":"9223372036854775807"} 2023-03-29 03:55:15,705 INFO [PEWorker-5] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-03-29 03:55:15,707 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(130): Finished pid=34, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-29 03:55:15,707 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=34, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 34 msec 2023-03-29 03:55:15,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=34 2023-03-29 03:55:15,776 INFO [Listener at localhost/35027] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 34 completed 2023-03-29 03:55:15,788 INFO [Listener at localhost/35027] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testGetAtomicity Thread=224 (was 226), OpenFileDescriptor=513 (was 512) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=367 (was 487), ProcessCount=169 (was 169), AvailableMemoryMB=6633 (was 6834) 2023-03-29 03:55:15,798 INFO [Listener at localhost/35027] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobScanAtomicity Thread=224, OpenFileDescriptor=513, MaxFileDescriptor=60000, SystemLoadAverage=367, ProcessCount=169, AvailableMemoryMB=6633 2023-03-29 03:55:15,799 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-29 03:55:15,799 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-29 03:55:15,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure2.ProcedureExecutor(1029): Stored pid=35, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-03-29 03:55:15,801 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-03-29 03:55:15,801 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 35 2023-03-29 03:55:15,802 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-03-29 03:55:15,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-03-29 03:55:15,803 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/.tmp/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67 2023-03-29 03:55:15,803 DEBUG [HFileArchiver-8] backup.HFileArchiver(153): Directory hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/.tmp/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67 empty. 2023-03-29 03:55:15,804 DEBUG [HFileArchiver-8] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/.tmp/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67 2023-03-29 03:55:15,804 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-29 03:55:15,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-03-29 03:55:16,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-03-29 03:55:16,210 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-29 03:55:16,211 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => a5c78075cc0f4fd1864c2f0339ad3d67, NAME => 'TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/.tmp 2023-03-29 03:55:16,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-03-29 03:55:16,617 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-29 03:55:16,617 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing a5c78075cc0f4fd1864c2f0339ad3d67, disabling compactions & flushes 2023-03-29 03:55:16,617 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:16,617 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:16,617 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. after waiting 0 ms 2023-03-29 03:55:16,617 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:16,617 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:16,617 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for a5c78075cc0f4fd1864c2f0339ad3d67: 2023-03-29 03:55:16,619 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-03-29 03:55:16,620 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680062116620"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680062116620"}]},"ts":"1680062116620"} 2023-03-29 03:55:16,621 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-03-29 03:55:16,621 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-03-29 03:55:16,622 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680062116621"}]},"ts":"1680062116621"} 2023-03-29 03:55:16,622 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-03-29 03:55:16,626 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=a5c78075cc0f4fd1864c2f0339ad3d67, ASSIGN}] 2023-03-29 03:55:16,627 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=a5c78075cc0f4fd1864c2f0339ad3d67, ASSIGN 2023-03-29 03:55:16,628 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=a5c78075cc0f4fd1864c2f0339ad3d67, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,35599,1680062018534; forceNewPlan=false, retain=false 2023-03-29 03:55:16,779 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=a5c78075cc0f4fd1864c2f0339ad3d67, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:16,779 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680062116779"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680062116779"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680062116779"}]},"ts":"1680062116779"} 2023-03-29 03:55:16,780 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=37, ppid=36, state=RUNNABLE; OpenRegionProcedure a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534}] 2023-03-29 03:55:16,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-03-29 03:55:16,935 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:16,935 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => a5c78075cc0f4fd1864c2f0339ad3d67, NAME => 'TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67.', STARTKEY => '', ENDKEY => ''} 2023-03-29 03:55:16,935 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees a5c78075cc0f4fd1864c2f0339ad3d67 2023-03-29 03:55:16,935 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-29 03:55:16,935 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for a5c78075cc0f4fd1864c2f0339ad3d67 2023-03-29 03:55:16,935 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for a5c78075cc0f4fd1864c2f0339ad3d67 2023-03-29 03:55:16,936 INFO [StoreOpener-a5c78075cc0f4fd1864c2f0339ad3d67-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region a5c78075cc0f4fd1864c2f0339ad3d67 2023-03-29 03:55:16,937 DEBUG [StoreOpener-a5c78075cc0f4fd1864c2f0339ad3d67-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A 2023-03-29 03:55:16,937 DEBUG [StoreOpener-a5c78075cc0f4fd1864c2f0339ad3d67-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A 2023-03-29 03:55:16,937 INFO [StoreOpener-a5c78075cc0f4fd1864c2f0339ad3d67-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-29 03:55:16,937 INFO [StoreOpener-a5c78075cc0f4fd1864c2f0339ad3d67-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a5c78075cc0f4fd1864c2f0339ad3d67 columnFamilyName A 2023-03-29 03:55:16,938 INFO [StoreOpener-a5c78075cc0f4fd1864c2f0339ad3d67-1] regionserver.HStore(310): Store=a5c78075cc0f4fd1864c2f0339ad3d67/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-29 03:55:16,938 INFO [StoreOpener-a5c78075cc0f4fd1864c2f0339ad3d67-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region a5c78075cc0f4fd1864c2f0339ad3d67 2023-03-29 03:55:16,939 DEBUG [StoreOpener-a5c78075cc0f4fd1864c2f0339ad3d67-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B 2023-03-29 03:55:16,939 DEBUG [StoreOpener-a5c78075cc0f4fd1864c2f0339ad3d67-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B 2023-03-29 03:55:16,939 INFO [StoreOpener-a5c78075cc0f4fd1864c2f0339ad3d67-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-29 03:55:16,939 INFO [StoreOpener-a5c78075cc0f4fd1864c2f0339ad3d67-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a5c78075cc0f4fd1864c2f0339ad3d67 columnFamilyName B 2023-03-29 03:55:16,940 INFO [StoreOpener-a5c78075cc0f4fd1864c2f0339ad3d67-1] regionserver.HStore(310): Store=a5c78075cc0f4fd1864c2f0339ad3d67/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-29 03:55:16,940 INFO [StoreOpener-a5c78075cc0f4fd1864c2f0339ad3d67-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region a5c78075cc0f4fd1864c2f0339ad3d67 2023-03-29 03:55:16,940 DEBUG [StoreOpener-a5c78075cc0f4fd1864c2f0339ad3d67-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C 2023-03-29 03:55:16,940 DEBUG [StoreOpener-a5c78075cc0f4fd1864c2f0339ad3d67-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C 2023-03-29 03:55:16,941 INFO [StoreOpener-a5c78075cc0f4fd1864c2f0339ad3d67-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-29 03:55:16,941 INFO [StoreOpener-a5c78075cc0f4fd1864c2f0339ad3d67-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a5c78075cc0f4fd1864c2f0339ad3d67 columnFamilyName C 2023-03-29 03:55:16,941 INFO [StoreOpener-a5c78075cc0f4fd1864c2f0339ad3d67-1] regionserver.HStore(310): Store=a5c78075cc0f4fd1864c2f0339ad3d67/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-29 03:55:16,941 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:16,942 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67 2023-03-29 03:55:16,942 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67 2023-03-29 03:55:16,943 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-29 03:55:16,944 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for a5c78075cc0f4fd1864c2f0339ad3d67 2023-03-29 03:55:16,945 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-29 03:55:16,946 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened a5c78075cc0f4fd1864c2f0339ad3d67; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72965696, jitterRate=0.08727359771728516}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-29 03:55:16,946 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for a5c78075cc0f4fd1864c2f0339ad3d67: 2023-03-29 03:55:16,946 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67., pid=37, masterSystemTime=1680062116932 2023-03-29 03:55:16,947 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:16,947 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:16,948 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=a5c78075cc0f4fd1864c2f0339ad3d67, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:16,948 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680062116948"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1680062116948"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680062116948"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680062116948"}]},"ts":"1680062116948"} 2023-03-29 03:55:16,950 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=37, resume processing ppid=36 2023-03-29 03:55:16,950 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=37, ppid=36, state=SUCCESS; OpenRegionProcedure a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 in 169 msec 2023-03-29 03:55:16,951 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=36, resume processing ppid=35 2023-03-29 03:55:16,951 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=36, ppid=35, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=a5c78075cc0f4fd1864c2f0339ad3d67, ASSIGN in 324 msec 2023-03-29 03:55:16,952 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-03-29 03:55:16,952 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680062116952"}]},"ts":"1680062116952"} 2023-03-29 03:55:16,953 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-03-29 03:55:16,955 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-03-29 03:55:16,956 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=35, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1560 sec 2023-03-29 03:55:17,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-03-29 03:55:17,906 INFO [Listener at localhost/35027] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 35 completed 2023-03-29 03:55:17,907 DEBUG [Listener at localhost/35027] zookeeper.ReadOnlyZKClient(139): Connect 0x1a5ab074 to 127.0.0.1:52103 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-29 03:55:17,912 DEBUG [Listener at localhost/35027] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@bb27fd4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-29 03:55:17,913 DEBUG [Listener at localhost/35027] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-29 03:55:17,915 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51922, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-29 03:55:17,916 DEBUG [Listener at localhost/35027] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-03-29 03:55:17,917 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:42564, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-03-29 03:55:17,918 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-29 03:55:17,918 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.HMaster$12(2454): Client=jenkins//172.31.14.131 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-29 03:55:17,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure2.ProcedureExecutor(1029): Stored pid=38, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-03-29 03:55:17,927 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-03-29 03:55:17,927 DEBUG [PEWorker-4] util.FSTableDescriptors(519): Deleted hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-29 03:55:17,928 INFO [PEWorker-4] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-03-29 03:55:17,929 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=39, ppid=38, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-03-29 03:55:17,931 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=a5c78075cc0f4fd1864c2f0339ad3d67, REOPEN/MOVE}] 2023-03-29 03:55:17,932 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=a5c78075cc0f4fd1864c2f0339ad3d67, REOPEN/MOVE 2023-03-29 03:55:17,932 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=a5c78075cc0f4fd1864c2f0339ad3d67, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:17,933 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680062117932"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680062117932"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680062117932"}]},"ts":"1680062117932"} 2023-03-29 03:55:17,933 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=41, ppid=40, state=RUNNABLE; CloseRegionProcedure a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534}] 2023-03-29 03:55:18,085 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close a5c78075cc0f4fd1864c2f0339ad3d67 2023-03-29 03:55:18,086 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing a5c78075cc0f4fd1864c2f0339ad3d67, disabling compactions & flushes 2023-03-29 03:55:18,086 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:18,086 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:18,086 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. after waiting 0 ms 2023-03-29 03:55:18,086 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:18,089 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-03-29 03:55:18,089 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:18,090 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for a5c78075cc0f4fd1864c2f0339ad3d67: 2023-03-29 03:55:18,090 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: a5c78075cc0f4fd1864c2f0339ad3d67 to self. 2023-03-29 03:55:18,091 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed a5c78075cc0f4fd1864c2f0339ad3d67 2023-03-29 03:55:18,091 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=a5c78075cc0f4fd1864c2f0339ad3d67, regionState=CLOSED 2023-03-29 03:55:18,091 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680062118091"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680062118091"}]},"ts":"1680062118091"} 2023-03-29 03:55:18,093 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=41, resume processing ppid=40 2023-03-29 03:55:18,093 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=41, ppid=40, state=SUCCESS; CloseRegionProcedure a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 in 159 msec 2023-03-29 03:55:18,094 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=a5c78075cc0f4fd1864c2f0339ad3d67, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,35599,1680062018534; forceNewPlan=false, retain=true 2023-03-29 03:55:18,244 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=a5c78075cc0f4fd1864c2f0339ad3d67, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:18,244 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680062118244"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680062118244"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680062118244"}]},"ts":"1680062118244"} 2023-03-29 03:55:18,245 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=42, ppid=40, state=RUNNABLE; OpenRegionProcedure a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534}] 2023-03-29 03:55:18,400 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:18,400 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => a5c78075cc0f4fd1864c2f0339ad3d67, NAME => 'TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67.', STARTKEY => '', ENDKEY => ''} 2023-03-29 03:55:18,401 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees a5c78075cc0f4fd1864c2f0339ad3d67 2023-03-29 03:55:18,401 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-29 03:55:18,401 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for a5c78075cc0f4fd1864c2f0339ad3d67 2023-03-29 03:55:18,401 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for a5c78075cc0f4fd1864c2f0339ad3d67 2023-03-29 03:55:18,402 INFO [StoreOpener-a5c78075cc0f4fd1864c2f0339ad3d67-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region a5c78075cc0f4fd1864c2f0339ad3d67 2023-03-29 03:55:18,403 DEBUG [StoreOpener-a5c78075cc0f4fd1864c2f0339ad3d67-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A 2023-03-29 03:55:18,403 DEBUG [StoreOpener-a5c78075cc0f4fd1864c2f0339ad3d67-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A 2023-03-29 03:55:18,403 INFO [StoreOpener-a5c78075cc0f4fd1864c2f0339ad3d67-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-29 03:55:18,403 INFO [StoreOpener-a5c78075cc0f4fd1864c2f0339ad3d67-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a5c78075cc0f4fd1864c2f0339ad3d67 columnFamilyName A 2023-03-29 03:55:18,405 INFO [StoreOpener-a5c78075cc0f4fd1864c2f0339ad3d67-1] regionserver.HStore(310): Store=a5c78075cc0f4fd1864c2f0339ad3d67/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-29 03:55:18,405 INFO [StoreOpener-a5c78075cc0f4fd1864c2f0339ad3d67-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region a5c78075cc0f4fd1864c2f0339ad3d67 2023-03-29 03:55:18,405 DEBUG [StoreOpener-a5c78075cc0f4fd1864c2f0339ad3d67-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B 2023-03-29 03:55:18,405 DEBUG [StoreOpener-a5c78075cc0f4fd1864c2f0339ad3d67-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B 2023-03-29 03:55:18,406 INFO [StoreOpener-a5c78075cc0f4fd1864c2f0339ad3d67-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-29 03:55:18,406 INFO [StoreOpener-a5c78075cc0f4fd1864c2f0339ad3d67-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a5c78075cc0f4fd1864c2f0339ad3d67 columnFamilyName B 2023-03-29 03:55:18,406 INFO [StoreOpener-a5c78075cc0f4fd1864c2f0339ad3d67-1] regionserver.HStore(310): Store=a5c78075cc0f4fd1864c2f0339ad3d67/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-29 03:55:18,406 INFO [StoreOpener-a5c78075cc0f4fd1864c2f0339ad3d67-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region a5c78075cc0f4fd1864c2f0339ad3d67 2023-03-29 03:55:18,407 DEBUG [StoreOpener-a5c78075cc0f4fd1864c2f0339ad3d67-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C 2023-03-29 03:55:18,407 DEBUG [StoreOpener-a5c78075cc0f4fd1864c2f0339ad3d67-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C 2023-03-29 03:55:18,407 INFO [StoreOpener-a5c78075cc0f4fd1864c2f0339ad3d67-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-29 03:55:18,407 INFO [StoreOpener-a5c78075cc0f4fd1864c2f0339ad3d67-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a5c78075cc0f4fd1864c2f0339ad3d67 columnFamilyName C 2023-03-29 03:55:18,407 INFO [StoreOpener-a5c78075cc0f4fd1864c2f0339ad3d67-1] regionserver.HStore(310): Store=a5c78075cc0f4fd1864c2f0339ad3d67/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-29 03:55:18,407 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:18,408 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67 2023-03-29 03:55:18,409 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67 2023-03-29 03:55:18,410 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-29 03:55:18,411 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for a5c78075cc0f4fd1864c2f0339ad3d67 2023-03-29 03:55:18,412 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened a5c78075cc0f4fd1864c2f0339ad3d67; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72821885, jitterRate=0.08513064682483673}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-29 03:55:18,412 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for a5c78075cc0f4fd1864c2f0339ad3d67: 2023-03-29 03:55:18,413 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67., pid=42, masterSystemTime=1680062118397 2023-03-29 03:55:18,414 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:18,414 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:18,414 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=a5c78075cc0f4fd1864c2f0339ad3d67, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:18,414 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680062118414"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1680062118414"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680062118414"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680062118414"}]},"ts":"1680062118414"} 2023-03-29 03:55:18,416 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=42, resume processing ppid=40 2023-03-29 03:55:18,416 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=42, ppid=40, state=SUCCESS; OpenRegionProcedure a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 in 170 msec 2023-03-29 03:55:18,417 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=40, resume processing ppid=39 2023-03-29 03:55:18,417 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=40, ppid=39, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=a5c78075cc0f4fd1864c2f0339ad3d67, REOPEN/MOVE in 485 msec 2023-03-29 03:55:18,419 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=39, resume processing ppid=38 2023-03-29 03:55:18,419 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=39, ppid=38, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 489 msec 2023-03-29 03:55:18,421 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=38, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 502 msec 2023-03-29 03:55:18,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=38 2023-03-29 03:55:18,422 DEBUG [Listener at localhost/35027] zookeeper.ReadOnlyZKClient(139): Connect 0x0bcfc1e0 to 127.0.0.1:52103 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-29 03:55:18,426 DEBUG [Listener at localhost/35027] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@445919fd, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-29 03:55:18,427 DEBUG [Listener at localhost/35027] zookeeper.ReadOnlyZKClient(139): Connect 0x38341134 to 127.0.0.1:52103 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-29 03:55:18,430 DEBUG [Listener at localhost/35027] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4d010713, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-29 03:55:18,430 DEBUG [Listener at localhost/35027] zookeeper.ReadOnlyZKClient(139): Connect 0x71b5058b to 127.0.0.1:52103 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-29 03:55:18,437 DEBUG [Listener at localhost/35027] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@69100738, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-29 03:55:18,438 DEBUG [Listener at localhost/35027] zookeeper.ReadOnlyZKClient(139): Connect 0x3620ee35 to 127.0.0.1:52103 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-29 03:55:18,441 DEBUG [Listener at localhost/35027] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6f603c27, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-29 03:55:18,441 DEBUG [Listener at localhost/35027] zookeeper.ReadOnlyZKClient(139): Connect 0x4fe1db6a to 127.0.0.1:52103 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-29 03:55:18,446 DEBUG [Listener at localhost/35027] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7e01d5e5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-29 03:55:18,446 DEBUG [Listener at localhost/35027] zookeeper.ReadOnlyZKClient(139): Connect 0x51ccd8b7 to 127.0.0.1:52103 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-29 03:55:18,450 DEBUG [Listener at localhost/35027] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@21128196, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-29 03:55:18,450 DEBUG [Listener at localhost/35027] zookeeper.ReadOnlyZKClient(139): Connect 0x68a0e21a to 127.0.0.1:52103 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-29 03:55:18,454 DEBUG [Listener at localhost/35027] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5f9e86d9, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-29 03:55:18,454 DEBUG [Listener at localhost/35027] zookeeper.ReadOnlyZKClient(139): Connect 0x1ec12b02 to 127.0.0.1:52103 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-29 03:55:18,459 DEBUG [Listener at localhost/35027] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@711ebaf2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-29 03:55:18,459 DEBUG [Listener at localhost/35027] zookeeper.ReadOnlyZKClient(139): Connect 0x1d7ba242 to 127.0.0.1:52103 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-29 03:55:18,463 DEBUG [Listener at localhost/35027] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@a7503d4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-29 03:55:18,463 DEBUG [Listener at localhost/35027] zookeeper.ReadOnlyZKClient(139): Connect 0x3e079da1 to 127.0.0.1:52103 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-29 03:55:18,467 DEBUG [Listener at localhost/35027] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7eae4191, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-29 03:55:18,471 DEBUG [hconnection-0x3535a4f9-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-29 03:55:18,472 DEBUG [hconnection-0x49114988-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-29 03:55:18,471 DEBUG [hconnection-0x111d228-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-29 03:55:18,474 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51928, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-29 03:55:18,475 DEBUG [hconnection-0x2c29380a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-29 03:55:18,475 DEBUG [hconnection-0x2bc07f6b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-29 03:55:18,475 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51946, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-29 03:55:18,475 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51938, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-29 03:55:18,476 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51956, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-29 03:55:18,482 DEBUG [hconnection-0x139980a0-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-29 03:55:18,482 DEBUG [hconnection-0x27373126-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-29 03:55:18,482 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51970, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-29 03:55:18,483 DEBUG [hconnection-0x1d809995-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-29 03:55:18,483 DEBUG [hconnection-0x6994c270-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-29 03:55:18,484 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:18,484 DEBUG [hconnection-0x1a9f2f8d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-29 03:55:18,486 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51990, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-29 03:55:18,486 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51994, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-29 03:55:18,486 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51980, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-29 03:55:18,488 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52010, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-29 03:55:18,488 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:51996, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-29 03:55:18,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:18,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:18,492 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:18,492 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:18,492 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:18,494 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:18,494 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:18,494 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:18,496 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:18,496 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:18,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:18,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:18,496 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:18,497 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:18,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:18,497 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:18,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:18,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:18,497 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:18,498 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:18,498 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:18,498 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:18,498 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:18,498 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:18,500 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:18,500 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:18,500 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:18,501 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. started... 2023-03-29 03:55:18,501 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a5c78075cc0f4fd1864c2f0339ad3d67 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-03-29 03:55:18,502 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5c78075cc0f4fd1864c2f0339ad3d67, store=A 2023-03-29 03:55:18,502 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:55:18,502 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5c78075cc0f4fd1864c2f0339ad3d67, store=B 2023-03-29 03:55:18,502 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:55:18,502 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5c78075cc0f4fd1864c2f0339ad3d67, store=C 2023-03-29 03:55:18,502 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:55:18,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on a5c78075cc0f4fd1864c2f0339ad3d67 2023-03-29 03:55:18,504 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. as already flushing 2023-03-29 03:55:18,521 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1680062118510, max timestamp=1680062118519Num uniques 859; 2023-03-29 03:55:18,521 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-29 03:55:18,522 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680062118510, max timestamp=1680062118520Num uniques 859; 2023-03-29 03:55:18,522 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-29 03:55:18,522 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680062118510, max timestamp=1680062118519Num uniques 859; 2023-03-29 03:55:18,522 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-29 03:55:18,540 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1680062118530, max timestamp=1680062118538Num uniques 1068; 2023-03-29 03:55:18,540 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=12268, suffix heap size=139040,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=268 2023-03-29 03:55:18,541 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1680062118530, max timestamp=1680062118538Num uniques 1018; 2023-03-29 03:55:18,541 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-03-29 03:55:18,541 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1680062118530, max timestamp=1680062118538Num uniques 1018; 2023-03-29 03:55:18,541 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-03-29 03:55:18,549 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:18,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51938 deadline: 1680062178548, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:18,550 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:18,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51994 deadline: 1680062178549, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:18,550 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:18,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51946 deadline: 1680062178549, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:18,555 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:18,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51970 deadline: 1680062178549, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:18,555 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:18,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51990 deadline: 1680062178549, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:18,652 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:18,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51938 deadline: 1680062178650, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:18,653 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:18,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51994 deadline: 1680062178651, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:18,653 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:18,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51946 deadline: 1680062178651, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:18,662 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:18,662 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:18,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51970 deadline: 1680062178656, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:18,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51990 deadline: 1680062178656, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:18,858 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:18,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51946 deadline: 1680062178854, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:18,858 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:18,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51938 deadline: 1680062178855, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:18,858 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:18,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51994 deadline: 1680062178855, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:18,864 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:18,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51970 deadline: 1680062178864, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:18,869 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:18,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51990 deadline: 1680062178865, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:18,924 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032940d4169d8e10421785da2f748d2b4217 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032940d4169d8e10421785da2f748d2b4217 2023-03-29 03:55:19,162 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:19,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51946 deadline: 1680062179159, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:19,163 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:19,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51938 deadline: 1680062179161, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:19,163 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:19,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51994 deadline: 1680062179161, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:19,169 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:19,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51970 deadline: 1680062179166, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:19,173 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:19,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51990 deadline: 1680062179171, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:19,329 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=12, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/A/413b333ae3934688bf785f28dd43be25 2023-03-29 03:55:19,341 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=12 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/B/c79a2874dd184bcba8eb67a074c63a88 2023-03-29 03:55:19,669 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:19,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51946 deadline: 1680062179665, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:19,669 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:19,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51938 deadline: 1680062179667, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:19,670 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:19,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51994 deadline: 1680062179668, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:19,679 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:19,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51970 deadline: 1680062179673, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:19,684 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:19,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51990 deadline: 1680062179678, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:19,751 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=12 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/C/7522e7a58f7749b4a733fc2e1bae07af 2023-03-29 03:55:19,754 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/A/413b333ae3934688bf785f28dd43be25 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/413b333ae3934688bf785f28dd43be25 2023-03-29 03:55:19,758 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/413b333ae3934688bf785f28dd43be25, entries=150, sequenceid=12, filesize=25.1 K 2023-03-29 03:55:19,758 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/B/c79a2874dd184bcba8eb67a074c63a88 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/c79a2874dd184bcba8eb67a074c63a88 2023-03-29 03:55:19,762 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/c79a2874dd184bcba8eb67a074c63a88, entries=150, sequenceid=12, filesize=11.6 K 2023-03-29 03:55:19,762 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/C/7522e7a58f7749b4a733fc2e1bae07af as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/7522e7a58f7749b4a733fc2e1bae07af 2023-03-29 03:55:19,765 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/7522e7a58f7749b4a733fc2e1bae07af, entries=150, sequenceid=12, filesize=11.6 K 2023-03-29 03:55:19,766 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=147.60 KB/151140 for a5c78075cc0f4fd1864c2f0339ad3d67 in 1265ms, sequenceid=12, compaction requested=false 2023-03-29 03:55:19,766 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-03-29 03:55:19,767 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a5c78075cc0f4fd1864c2f0339ad3d67: 2023-03-29 03:55:19,767 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:19,767 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:19,767 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:19,767 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:19,767 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:19,767 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:19,769 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:19,769 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:19,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:19,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:19,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:19,769 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:19,769 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:19,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:19,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:19,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:19,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:19,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:19,770 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:19,771 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6346d648[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:19,771 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:19,771 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:19,772 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:19,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:19,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:19,772 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:19,772 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:19,772 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:19,772 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:19,772 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:19,773 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:19,773 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:19,773 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:19,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:19,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:19,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:19,773 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:19,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:19,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:19,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:19,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:19,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:19,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:19,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:19,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:19,776 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:19,776 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:19,776 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:19,776 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:19,776 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:19,776 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:19,780 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:19,780 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:19,780 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:19,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:19,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:19,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:19,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:19,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:19,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:19,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:19,781 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:19,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:19,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:19,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:19,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:19,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:19,782 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:19,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:19,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:19,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:19,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:19,785 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:19,785 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:19,785 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:19,785 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:19,785 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:19,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:19,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:19,785 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:19,785 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:19,785 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:19,786 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:19,786 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:19,786 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:19,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:19,786 DEBUG [Thread-2498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:19,786 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:55:19,786 DEBUG [Thread-2498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:19,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:19,786 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:19,886 DEBUG [Thread-2498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:19,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:19,888 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:19,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:19,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:19,890 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:19,890 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:19,890 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:19,890 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:19,891 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:19,891 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:19,892 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:19,892 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:19,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:19,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:19,892 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:19,892 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:19,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:19,893 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:19,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:19,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:19,893 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:19,893 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:19,893 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:19,894 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:19,894 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:19,894 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:19,894 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:19,894 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:19,894 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:19,894 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. started... 2023-03-29 03:55:19,894 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a5c78075cc0f4fd1864c2f0339ad3d67 3/3 column families, dataSize=147.60 KB heapSize=414.61 KB 2023-03-29 03:55:19,894 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5c78075cc0f4fd1864c2f0339ad3d67, store=A 2023-03-29 03:55:19,894 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-29 03:55:19,894 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5c78075cc0f4fd1864c2f0339ad3d67, store=B 2023-03-29 03:55:19,894 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-29 03:55:19,894 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5c78075cc0f4fd1864c2f0339ad3d67, store=C 2023-03-29 03:55:19,894 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-29 03:55:20,311 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230329fe3c99e3fb914c16beee4e4f697614a4 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230329fe3c99e3fb914c16beee4e4f697614a4 2023-03-29 03:55:20,640 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-03-29 03:55:20,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on a5c78075cc0f4fd1864c2f0339ad3d67 2023-03-29 03:55:20,679 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. as already flushing 2023-03-29 03:55:20,689 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:20,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51970 deadline: 1680062180683, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:20,694 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:20,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51946 deadline: 1680062180687, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:20,694 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:20,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51938 deadline: 1680062180688, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:20,694 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:20,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51994 deadline: 1680062180689, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:20,699 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:20,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51990 deadline: 1680062180691, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:20,717 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=66, memsize=49.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/A/99006d66cc3e432b8e2abc2436babbc6 2023-03-29 03:55:20,734 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=66 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/B/15a8b139d38f457d9284e7e2e260d532 2023-03-29 03:55:20,799 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:20,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51946 deadline: 1680062180795, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:20,799 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:20,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51938 deadline: 1680062180795, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:20,799 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:20,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51994 deadline: 1680062180795, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:21,004 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:21,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51938 deadline: 1680062181000, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:21,004 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:21,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51994 deadline: 1680062181000, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:21,004 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:21,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51946 deadline: 1680062181001, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:21,146 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=66 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/C/528af543643c4eed9cf12d5c2aa68c5e 2023-03-29 03:55:21,151 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/A/99006d66cc3e432b8e2abc2436babbc6 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/99006d66cc3e432b8e2abc2436babbc6 2023-03-29 03:55:21,157 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/99006d66cc3e432b8e2abc2436babbc6, entries=150, sequenceid=66, filesize=25.1 K 2023-03-29 03:55:21,159 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/B/15a8b139d38f457d9284e7e2e260d532 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/15a8b139d38f457d9284e7e2e260d532 2023-03-29 03:55:21,164 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/15a8b139d38f457d9284e7e2e260d532, entries=150, sequenceid=66, filesize=11.6 K 2023-03-29 03:55:21,165 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/C/528af543643c4eed9cf12d5c2aa68c5e as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/528af543643c4eed9cf12d5c2aa68c5e 2023-03-29 03:55:21,169 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/528af543643c4eed9cf12d5c2aa68c5e, entries=150, sequenceid=66, filesize=11.6 K 2023-03-29 03:55:21,169 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~414.56 KB/424512, currentSize=33.54 KB/34350 for a5c78075cc0f4fd1864c2f0339ad3d67 in 1275ms, sequenceid=66, compaction requested=false 2023-03-29 03:55:21,170 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a5c78075cc0f4fd1864c2f0339ad3d67: 2023-03-29 03:55:21,170 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:21,170 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:21,170 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:21,170 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:21,170 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:21,170 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:21,171 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:21,171 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:21,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:21,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:21,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:21,172 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:21,172 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:21,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:21,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:21,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:21,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:21,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:21,174 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:21,174 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@235b0bd8[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:21,174 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:21,174 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:21,176 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:21,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:21,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:21,176 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:21,176 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:21,176 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:21,176 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:21,176 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:21,180 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:21,180 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:21,180 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:21,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:21,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:21,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:21,180 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:21,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:21,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:21,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:21,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:21,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:21,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:21,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:21,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:21,185 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:21,185 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:21,185 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:21,185 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:21,185 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:21,185 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:21,186 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:21,186 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:21,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:21,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:21,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:21,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:21,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:21,186 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:21,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:21,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:21,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:21,187 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:21,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:21,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:21,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:21,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:21,188 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:21,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:21,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:21,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:21,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:21,192 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:21,192 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:21,192 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:21,192 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:21,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:21,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:21,192 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:21,192 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:21,192 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:21,192 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:21,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:21,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:21,192 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:21,192 DEBUG [Thread-2498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:21,192 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:21,192 DEBUG [Thread-2498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:21,192 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:21,193 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:21,293 DEBUG [Thread-2498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:21,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:21,295 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:21,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:21,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:21,297 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:21,297 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:21,297 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:21,297 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:21,298 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:21,298 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:21,299 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:21,299 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:21,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:21,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:21,299 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:21,299 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:21,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:21,300 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:21,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:21,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:21,301 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:21,301 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:21,301 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:21,301 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:21,301 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:21,301 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:21,301 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:21,301 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:21,301 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:21,301 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. started... 2023-03-29 03:55:21,301 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a5c78075cc0f4fd1864c2f0339ad3d67 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-03-29 03:55:21,302 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5c78075cc0f4fd1864c2f0339ad3d67, store=A 2023-03-29 03:55:21,302 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:55:21,302 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5c78075cc0f4fd1864c2f0339ad3d67, store=B 2023-03-29 03:55:21,302 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:55:21,302 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5c78075cc0f4fd1864c2f0339ad3d67, store=C 2023-03-29 03:55:21,302 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:55:21,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on a5c78075cc0f4fd1864c2f0339ad3d67 2023-03-29 03:55:21,325 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. as already flushing 2023-03-29 03:55:21,382 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680062121366, max timestamp=1680062121381Num uniques 859; 2023-03-29 03:55:21,382 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-29 03:55:21,382 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680062121366, max timestamp=1680062121381Num uniques 859; 2023-03-29 03:55:21,383 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-29 03:55:21,383 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680062121366, max timestamp=1680062121381Num uniques 859; 2023-03-29 03:55:21,383 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-29 03:55:21,418 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:21,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51938 deadline: 1680062181412, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:21,418 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:21,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51994 deadline: 1680062181415, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:21,421 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:21,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51946 deadline: 1680062181415, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:21,521 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:21,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51938 deadline: 1680062181519, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:21,521 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:21,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51994 deadline: 1680062181519, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:21,524 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:21,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51946 deadline: 1680062181522, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:21,714 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230329f1f5b27850674fb8bf77f02be0ae1fb1 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230329f1f5b27850674fb8bf77f02be0ae1fb1 2023-03-29 03:55:21,727 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:21,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51938 deadline: 1680062181722, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:21,727 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:21,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51994 deadline: 1680062181723, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:21,731 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:21,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51946 deadline: 1680062181726, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:22,035 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:22,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51938 deadline: 1680062182028, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:22,035 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:22,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51994 deadline: 1680062182028, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:22,037 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:22,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51946 deadline: 1680062182032, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:22,119 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=74, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/A/cafee5e0813f4a2da3cd17f07119e801 2023-03-29 03:55:22,128 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=74 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/B/5052efffd2a84a8d9080bcba50d3a88b 2023-03-29 03:55:22,536 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=74 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/C/dc646a8263e54b96b5315974d304b5af 2023-03-29 03:55:22,541 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/A/cafee5e0813f4a2da3cd17f07119e801 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/cafee5e0813f4a2da3cd17f07119e801 2023-03-29 03:55:22,544 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:22,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51994 deadline: 1680062182537, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:22,544 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:22,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51938 deadline: 1680062182538, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:22,544 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:22,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51946 deadline: 1680062182538, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:22,546 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/cafee5e0813f4a2da3cd17f07119e801, entries=100, sequenceid=74, filesize=18.4 K 2023-03-29 03:55:22,547 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/B/5052efffd2a84a8d9080bcba50d3a88b as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/5052efffd2a84a8d9080bcba50d3a88b 2023-03-29 03:55:22,554 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/5052efffd2a84a8d9080bcba50d3a88b, entries=100, sequenceid=74, filesize=9.3 K 2023-03-29 03:55:22,554 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/C/dc646a8263e54b96b5315974d304b5af as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/dc646a8263e54b96b5315974d304b5af 2023-03-29 03:55:22,559 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/dc646a8263e54b96b5315974d304b5af, entries=100, sequenceid=74, filesize=9.3 K 2023-03-29 03:55:22,560 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=154.31 KB/158010 for a5c78075cc0f4fd1864c2f0339ad3d67 in 1259ms, sequenceid=74, compaction requested=true 2023-03-29 03:55:22,560 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a5c78075cc0f4fd1864c2f0339ad3d67: 2023-03-29 03:55:22,560 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:22,560 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:22,560 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:22,560 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:22,560 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:22,560 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:22,562 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:22,562 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:22,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:22,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:22,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:22,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:22,563 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:22,563 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:22,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:22,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:22,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:22,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:22,564 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:22,564 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d15d937[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:22,564 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:22,564 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:22,565 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:22,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:22,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:22,566 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:22,566 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:22,566 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:22,566 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:22,566 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:22,568 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:22,568 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:22,568 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:22,568 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:22,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:22,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:22,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:22,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:22,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:22,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:22,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:22,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:22,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:22,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:22,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:22,570 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:22,570 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:22,570 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:22,570 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:22,570 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:22,570 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:22,572 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:22,572 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:22,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:22,572 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:22,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:22,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:22,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:22,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:22,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:22,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:22,573 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:22,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:22,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:22,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:22,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:22,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:22,574 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:22,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:22,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:22,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:22,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:22,576 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:22,576 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:22,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:22,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:22,576 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:22,577 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:22,577 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:22,577 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:22,577 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:22,577 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:22,576 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:22,577 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:22,576 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:22,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:22,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:22,576 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:22,577 DEBUG [Thread-2498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:22,577 DEBUG [Thread-2498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:22,677 DEBUG [Thread-2498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:22,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:22,680 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:22,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:22,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:22,682 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:22,682 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:22,682 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:22,682 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:22,682 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:22,682 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:22,684 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:22,684 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:22,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:22,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:22,684 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:22,684 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:22,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:22,684 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:22,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:22,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:22,685 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:22,685 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:22,685 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:22,685 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:22,685 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:22,685 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:22,685 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:22,685 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:22,685 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:22,685 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. started... 2023-03-29 03:55:22,686 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a5c78075cc0f4fd1864c2f0339ad3d67 3/3 column families, dataSize=154.31 KB heapSize=430.69 KB 2023-03-29 03:55:22,686 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5c78075cc0f4fd1864c2f0339ad3d67, store=A 2023-03-29 03:55:22,686 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-29 03:55:22,686 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5c78075cc0f4fd1864c2f0339ad3d67, store=B 2023-03-29 03:55:22,686 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-29 03:55:22,686 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5c78075cc0f4fd1864c2f0339ad3d67, store=C 2023-03-29 03:55:22,686 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-29 03:55:22,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on a5c78075cc0f4fd1864c2f0339ad3d67 2023-03-29 03:55:22,705 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. as already flushing 2023-03-29 03:55:22,725 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:22,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51990 deadline: 1680062182721, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:22,732 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:22,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51970 deadline: 1680062182722, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:22,830 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:22,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51990 deadline: 1680062182827, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:22,836 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:22,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51970 deadline: 1680062182833, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:23,036 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:23,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51990 deadline: 1680062183031, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:23,043 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:23,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51970 deadline: 1680062183038, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:23,099 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032987b62070eed243d7944dd28f8e2f2cff to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032987b62070eed243d7944dd28f8e2f2cff 2023-03-29 03:55:23,339 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:23,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51990 deadline: 1680062183338, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:23,345 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:23,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51970 deadline: 1680062183344, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:23,507 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=111, memsize=51.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/A/6ee45ee7b9204e39933bc2b40c5a9b19 2023-03-29 03:55:23,551 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:23,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51994 deadline: 1680062183546, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:23,551 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:23,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51946 deadline: 1680062183547, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:23,553 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:23,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51938 deadline: 1680062183548, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:23,845 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:23,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51990 deadline: 1680062183841, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:23,852 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:23,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51970 deadline: 1680062183850, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:23,918 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=111 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/B/13fdff8451574686ab05e509355583bf 2023-03-29 03:55:23,927 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=111 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/C/eda0e120d6fa4644820b8f8b43605329 2023-03-29 03:55:23,931 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/A/6ee45ee7b9204e39933bc2b40c5a9b19 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/6ee45ee7b9204e39933bc2b40c5a9b19 2023-03-29 03:55:23,934 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/6ee45ee7b9204e39933bc2b40c5a9b19, entries=150, sequenceid=111, filesize=25.1 K 2023-03-29 03:55:23,935 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/B/13fdff8451574686ab05e509355583bf as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/13fdff8451574686ab05e509355583bf 2023-03-29 03:55:23,938 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/13fdff8451574686ab05e509355583bf, entries=150, sequenceid=111, filesize=11.6 K 2023-03-29 03:55:23,939 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/C/eda0e120d6fa4644820b8f8b43605329 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/eda0e120d6fa4644820b8f8b43605329 2023-03-29 03:55:23,942 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/eda0e120d6fa4644820b8f8b43605329, entries=150, sequenceid=111, filesize=11.6 K 2023-03-29 03:55:23,943 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~154.31 KB/158010, heapSize ~430.64 KB/440976, currentSize=33.54 KB/34350 for a5c78075cc0f4fd1864c2f0339ad3d67 in 1257ms, sequenceid=111, compaction requested=true 2023-03-29 03:55:23,943 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a5c78075cc0f4fd1864c2f0339ad3d67: 2023-03-29 03:55:23,943 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:23,943 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:23,943 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:23,943 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:23,943 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:23,943 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:23,945 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:23,945 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:23,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:23,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:23,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:23,945 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:23,945 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:23,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:23,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:23,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:23,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:23,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:23,947 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:23,947 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@22d027f7[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:23,947 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:23,947 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:23,948 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:23,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:23,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:23,948 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:23,948 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:23,948 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:23,950 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:23,950 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:23,951 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:23,951 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:23,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:23,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:23,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:23,951 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:23,952 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:23,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:23,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:23,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:23,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:23,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:23,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:23,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:23,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:23,955 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:23,955 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:23,955 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:23,955 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:23,955 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:23,955 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:23,957 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:23,957 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:23,957 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:23,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:23,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:23,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:23,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:23,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:23,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:23,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:23,957 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:23,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:23,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:23,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:23,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:23,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:23,959 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:23,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:23,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:23,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:23,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:23,961 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:23,962 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:23,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:23,962 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:23,962 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:23,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:23,962 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:23,962 DEBUG [Thread-2498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:23,962 DEBUG [Thread-2498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:23,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:23,962 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:55:23,962 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:23,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:23,962 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:23,962 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:23,962 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:23,962 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:23,962 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:23,962 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:24,062 DEBUG [Thread-2498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:24,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:24,064 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:24,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:24,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:24,066 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:24,066 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:24,066 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:24,066 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:24,066 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:24,066 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:24,068 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:24,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:24,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:24,068 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:24,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:24,068 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:24,068 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:24,069 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:24,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:24,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:24,069 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:24,069 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:24,069 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:24,069 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:24,069 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:24,069 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:24,070 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:24,070 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:24,070 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:24,070 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. started... 2023-03-29 03:55:24,070 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a5c78075cc0f4fd1864c2f0339ad3d67 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-03-29 03:55:24,070 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5c78075cc0f4fd1864c2f0339ad3d67, store=A 2023-03-29 03:55:24,070 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:55:24,070 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5c78075cc0f4fd1864c2f0339ad3d67, store=B 2023-03-29 03:55:24,070 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:55:24,070 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5c78075cc0f4fd1864c2f0339ad3d67, store=C 2023-03-29 03:55:24,070 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:55:24,488 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230329f1cd578ad8d24ea0aba16fc90c0c7265 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230329f1cd578ad8d24ea0aba16fc90c0c7265 2023-03-29 03:55:24,493 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=119, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/A/b815d2bcbd3a487ba420d75dddbe1c6d 2023-03-29 03:55:24,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on a5c78075cc0f4fd1864c2f0339ad3d67 2023-03-29 03:55:24,859 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. as already flushing 2023-03-29 03:55:24,903 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=119 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/B/d45741edc43e4f80a0e4ad2e592dfc6a 2023-03-29 03:55:24,907 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680062124883, max timestamp=1680062124905Num uniques 859; 2023-03-29 03:55:24,907 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-29 03:55:24,907 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680062124883, max timestamp=1680062124905Num uniques 859; 2023-03-29 03:55:24,907 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-29 03:55:24,908 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680062124883, max timestamp=1680062124905Num uniques 859; 2023-03-29 03:55:24,908 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-29 03:55:24,914 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=119 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/C/ebbc1e200ac34ceba35bf0c8c97c8c8e 2023-03-29 03:55:24,918 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/A/b815d2bcbd3a487ba420d75dddbe1c6d as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/b815d2bcbd3a487ba420d75dddbe1c6d 2023-03-29 03:55:24,921 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/b815d2bcbd3a487ba420d75dddbe1c6d, entries=150, sequenceid=119, filesize=25.1 K 2023-03-29 03:55:24,922 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/B/d45741edc43e4f80a0e4ad2e592dfc6a as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/d45741edc43e4f80a0e4ad2e592dfc6a 2023-03-29 03:55:24,928 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/d45741edc43e4f80a0e4ad2e592dfc6a, entries=150, sequenceid=119, filesize=11.6 K 2023-03-29 03:55:24,929 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/C/ebbc1e200ac34ceba35bf0c8c97c8c8e as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/ebbc1e200ac34ceba35bf0c8c97c8c8e 2023-03-29 03:55:24,937 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/ebbc1e200ac34ceba35bf0c8c97c8c8e, entries=150, sequenceid=119, filesize=11.6 K 2023-03-29 03:55:24,938 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=87.22 KB/89310 for a5c78075cc0f4fd1864c2f0339ad3d67 in 868ms, sequenceid=119, compaction requested=true 2023-03-29 03:55:24,938 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a5c78075cc0f4fd1864c2f0339ad3d67: 2023-03-29 03:55:24,938 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:24,938 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:24,938 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:24,938 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:24,938 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:24,938 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:24,940 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:24,940 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:24,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:24,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:24,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:24,940 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:24,940 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:24,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:24,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:24,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:24,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:24,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:24,942 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:24,942 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@77714173[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:24,942 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:24,942 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:24,944 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:24,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:24,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on a5c78075cc0f4fd1864c2f0339ad3d67 2023-03-29 03:55:24,944 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a5c78075cc0f4fd1864c2f0339ad3d67 3/3 column families, dataSize=93.93 KB heapSize=260.77 KB 2023-03-29 03:55:24,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:24,944 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:24,944 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:24,944 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:24,945 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:24,945 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:24,945 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5c78075cc0f4fd1864c2f0339ad3d67, store=A 2023-03-29 03:55:24,945 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-29 03:55:24,945 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5c78075cc0f4fd1864c2f0339ad3d67, store=B 2023-03-29 03:55:24,945 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-29 03:55:24,945 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5c78075cc0f4fd1864c2f0339ad3d67, store=C 2023-03-29 03:55:24,945 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-29 03:55:24,946 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:24,946 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:24,946 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:24,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:24,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:24,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:24,946 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:24,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:24,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:24,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:24,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:24,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:24,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:24,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:24,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:24,951 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:24,951 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:24,952 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:24,952 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:24,952 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:24,952 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:24,953 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:24,953 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:24,953 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:24,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:24,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:24,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:24,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:24,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:24,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:24,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:24,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:24,954 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:24,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:24,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:24,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:24,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:24,955 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:24,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:24,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:24,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:24,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:24,960 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:24,960 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:24,960 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:24,960 DEBUG [Thread-2498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:24,960 DEBUG [Thread-2498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:24,960 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:24,960 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:24,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:24,961 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:24,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:24,961 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:24,961 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:24,961 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:24,961 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:24,961 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:24,961 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:24,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:24,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:24,968 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303292346cc3fd25d4e9a8f999affccf42486 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303292346cc3fd25d4e9a8f999affccf42486 2023-03-29 03:55:24,978 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=150, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/A/dd266c63ee75421183eefedfab4f43c9 2023-03-29 03:55:25,035 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:25,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51990 deadline: 1680062185033, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:25,046 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:25,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51970 deadline: 1680062185035, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:25,061 DEBUG [Thread-2498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:25,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:25,062 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:25,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:25,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:25,064 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:25,064 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:25,064 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:25,064 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:25,064 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:25,065 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:25,066 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:25,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:25,066 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:25,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:25,066 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:25,066 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:25,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:25,067 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:25,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:25,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:25,067 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:25,067 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:25,067 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:25,067 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:25,067 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:25,067 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:25,068 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:25,068 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:25,068 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:25,068 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. started... 2023-03-29 03:55:25,068 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. as already flushing 2023-03-29 03:55:25,068 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:25,068 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:25,068 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:25,068 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:25,068 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:25,068 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:25,069 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:25,069 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:25,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:25,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:25,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:25,070 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:25,070 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:25,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:25,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:25,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:25,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:25,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:25,071 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:25,071 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@c8f818e[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:25,071 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:25,071 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:25,073 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:25,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:25,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:25,073 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:25,073 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:25,073 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:25,075 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:25,075 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:25,075 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:25,076 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:25,076 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:25,076 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:25,077 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:25,077 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:25,077 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:25,077 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:25,077 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:25,078 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:25,078 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:25,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:25,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:25,078 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:25,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:25,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:25,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:25,079 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:25,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:25,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:25,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:25,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:25,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:25,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:25,081 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:25,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:25,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:25,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:25,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:25,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:25,084 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:25,084 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:25,084 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:25,084 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:25,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:25,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:25,084 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:25,084 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:25,084 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:25,084 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:25,084 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:25,084 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:25,084 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:25,084 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:25,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:25,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:25,084 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:55:25,084 DEBUG [Thread-2498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:25,085 DEBUG [Thread-2498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:25,143 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:25,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51990 deadline: 1680062185136, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:25,151 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:25,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51970 deadline: 1680062185147, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:25,185 DEBUG [Thread-2498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:25,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:25,186 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:25,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:25,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:25,188 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:25,188 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:25,188 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:25,188 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:25,188 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:25,188 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:25,189 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:25,189 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:25,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:25,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:25,190 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:25,190 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:25,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:25,190 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:25,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:25,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:25,191 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:25,191 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:25,191 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:25,191 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:25,191 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:25,191 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:25,191 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:25,191 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:25,191 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:25,191 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. started... 2023-03-29 03:55:25,191 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. as already flushing 2023-03-29 03:55:25,192 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:25,192 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:25,192 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:25,192 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:25,192 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:25,192 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:25,194 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:25,194 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:25,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:25,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:25,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:25,194 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:25,194 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:25,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:25,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:25,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:25,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:25,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:25,196 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:25,196 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3f602d49[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:25,196 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:25,196 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:25,197 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:25,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:25,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:25,197 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:25,197 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:25,197 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:25,197 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:25,197 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:25,199 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:25,199 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:25,199 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:25,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:25,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:25,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:25,199 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:25,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:25,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:25,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:25,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:25,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:25,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:25,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:25,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:25,201 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:25,201 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:25,201 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:25,201 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:25,202 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:25,202 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:25,203 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:25,203 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:25,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:25,203 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:25,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:25,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:25,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:25,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:25,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:25,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:25,204 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:25,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:25,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:25,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:25,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:25,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:25,205 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:25,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:25,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:25,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:25,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:25,209 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:25,209 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:25,209 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:25,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:25,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:25,209 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:25,209 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:25,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:25,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:25,210 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:25,210 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:25,210 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:25,210 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:25,210 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:25,210 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:25,210 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:25,211 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:55:25,211 DEBUG [Thread-2498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:25,211 DEBUG [Thread-2498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:25,311 DEBUG [Thread-2498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:25,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:25,313 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:25,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:25,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:25,314 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:25,314 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:25,314 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:25,314 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:25,315 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:25,315 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:25,316 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:25,316 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:25,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:25,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:25,316 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:25,316 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:25,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:25,317 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:25,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:25,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:25,317 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:25,317 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:25,317 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:25,317 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:25,317 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:25,317 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:25,317 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:25,317 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:25,317 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:25,318 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. started... 2023-03-29 03:55:25,318 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. as already flushing 2023-03-29 03:55:25,318 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:25,318 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:25,318 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:25,318 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:25,318 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:25,318 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:25,319 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:25,319 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:25,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:25,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:25,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:25,320 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:25,320 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:25,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:25,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:25,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:25,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:25,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:25,321 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:25,321 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c2c4acd[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:25,321 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:25,321 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:25,322 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:25,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:25,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:25,322 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:25,322 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:25,323 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:25,323 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:25,323 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:25,325 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:25,325 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:25,325 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:25,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:25,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:25,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:25,325 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:25,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:25,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:25,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:25,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:25,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:25,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:25,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:25,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:25,327 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:25,327 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:25,327 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:25,327 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:25,327 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:25,327 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:25,329 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:25,329 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:25,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:25,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:25,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:25,329 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:25,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:25,329 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:25,329 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:25,329 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:25,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:25,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:25,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:25,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:25,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:25,330 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:25,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:25,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:25,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:25,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:25,335 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:25,335 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:25,335 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:25,335 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:25,335 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:25,336 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:25,335 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:25,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:25,336 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:55:25,335 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:25,335 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:25,336 DEBUG [Thread-2498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:25,336 DEBUG [Thread-2498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:25,336 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:25,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:25,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:25,336 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:25,336 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:25,336 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:25,351 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:25,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51990 deadline: 1680062185345, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:25,358 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:25,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51970 deadline: 1680062185352, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:25,389 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=150 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/B/2314c1fead8842ccb2a38abfbd527a3d 2023-03-29 03:55:25,436 DEBUG [Thread-2498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:25,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:25,438 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:25,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:25,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:25,440 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:25,440 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:25,440 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:25,440 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:25,441 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:25,441 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:25,443 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:25,444 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:25,444 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:25,445 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:25,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:25,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:25,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:25,445 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:25,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:25,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:25,446 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:25,446 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:25,446 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:25,446 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:25,446 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:25,446 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:25,447 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:25,447 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:25,447 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:25,447 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. started... 2023-03-29 03:55:25,447 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. as already flushing 2023-03-29 03:55:25,447 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:25,447 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:25,447 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:25,447 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:25,447 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:25,447 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:25,448 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:25,448 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:25,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:25,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:25,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:25,449 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:25,449 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:25,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:25,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:25,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:25,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:25,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:25,450 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:25,450 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@61abeae2[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:25,450 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:25,450 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:25,451 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:25,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:25,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:25,451 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:25,451 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:25,452 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:25,452 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:25,452 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:25,453 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:25,453 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:25,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:25,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:25,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:25,453 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:25,453 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:25,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:25,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:25,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:25,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:25,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:25,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:25,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:25,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:25,455 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:25,455 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:25,455 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:25,455 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:25,455 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:25,455 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:25,458 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:25,458 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:25,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:25,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:25,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:25,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:25,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:25,458 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:25,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:25,459 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:25,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:25,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:25,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:25,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:25,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:25,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:25,460 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:25,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:25,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:25,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:25,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:25,462 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:25,462 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:25,462 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:25,462 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:25,462 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:25,462 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:25,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:25,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:25,462 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:25,462 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:25,462 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:55:25,462 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:25,462 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:25,462 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:25,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:25,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:25,462 DEBUG [Thread-2498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:25,462 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:25,463 DEBUG [Thread-2498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:25,563 DEBUG [Thread-2498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:25,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:25,564 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:25,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51946 deadline: 1680062185559, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:25,565 DEBUG [Thread-2494] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4149 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67., hostname=jenkins-hbase4.apache.org,35599,1680062018534, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-29 03:55:25,566 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:25,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:25,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:25,567 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:25,567 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:25,567 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:25,567 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:25,568 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:25,568 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:25,569 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:25,569 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:25,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:25,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:25,569 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:25,569 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:25,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:25,570 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:25,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:25,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:25,570 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:25,570 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:25,570 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:25,571 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:25,571 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:25,571 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:25,571 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:25,571 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:25,571 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:25,571 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. started... 2023-03-29 03:55:25,571 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. as already flushing 2023-03-29 03:55:25,571 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:25,571 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:25,571 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:25,571 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:25,571 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:25,571 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:25,572 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:25,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51994 deadline: 1680062185564, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:25,573 DEBUG [Thread-2496] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4158 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67., hostname=jenkins-hbase4.apache.org,35599,1680062018534, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-29 03:55:25,573 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:25,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:25,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:25,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:25,573 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:25,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:25,574 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:25,574 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:25,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:25,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:25,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:25,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:25,575 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:25,575 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:25,575 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@11d884a8[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:25,575 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:25,576 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:25,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51938 deadline: 1680062185570, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:25,577 DEBUG [Thread-2488] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4164 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67., hostname=jenkins-hbase4.apache.org,35599,1680062018534, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-29 03:55:25,579 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:25,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:25,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:25,579 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:25,579 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:25,579 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:25,579 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:25,579 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:25,580 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:25,580 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:25,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:25,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:25,581 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:25,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:25,581 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:25,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:25,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:25,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:25,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:25,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:25,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:25,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:25,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:25,583 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:25,583 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:25,583 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:25,583 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:25,583 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:25,583 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:25,584 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:25,584 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:25,584 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:25,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:25,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:25,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:25,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:25,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:25,585 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:25,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:25,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:25,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:25,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:25,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:25,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:25,586 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:25,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:25,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:25,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:25,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:25,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:25,588 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:25,588 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:25,588 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:25,588 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:25,588 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:25,588 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:25,588 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:25,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:25,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:25,588 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:25,588 DEBUG [Thread-2498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:25,589 DEBUG [Thread-2498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:25,588 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:25,588 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:55:25,588 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:25,589 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:25,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:25,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:25,589 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:25,655 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:25,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51990 deadline: 1680062185653, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:25,665 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:25,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51970 deadline: 1680062185661, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:25,689 DEBUG [Thread-2498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:25,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:25,691 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:25,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:25,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:25,692 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:25,692 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:25,692 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:25,692 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:25,692 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:25,692 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:25,695 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:25,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:25,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:25,695 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:25,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:25,695 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:25,695 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:25,695 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:25,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:25,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:25,696 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:25,696 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:25,696 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:25,696 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:25,696 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:25,696 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:25,696 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:25,696 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:25,696 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:25,696 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. started... 2023-03-29 03:55:25,696 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. as already flushing 2023-03-29 03:55:25,696 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:25,697 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:25,697 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:25,697 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:25,697 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:25,697 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:25,698 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:25,698 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:25,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:25,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:25,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:25,698 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:25,698 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:25,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:25,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:25,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:25,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:25,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:25,700 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:25,700 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f6ce9bf[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:25,700 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:25,700 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:25,701 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:25,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:25,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:25,701 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:25,701 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:25,702 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:25,702 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:25,702 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:25,703 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:25,703 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:25,703 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:25,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:25,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:25,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:25,703 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:25,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:25,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:25,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:25,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:25,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:25,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:25,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:25,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:25,705 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:25,705 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:25,705 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:25,705 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:25,705 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:25,705 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:25,708 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:25,708 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:25,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:25,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:25,708 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:25,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:25,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:25,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:25,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:25,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:25,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:25,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:25,709 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:25,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:25,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:25,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:25,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:25,710 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:25,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:25,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:25,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:25,712 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:25,712 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:25,712 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:55:25,712 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:25,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:25,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:25,712 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:25,712 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:25,712 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:25,712 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:25,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:25,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:25,712 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:25,713 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:25,713 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:25,713 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:25,713 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:25,713 DEBUG [Thread-2498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:25,713 DEBUG [Thread-2498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:25,799 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=150 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/C/45dba969a2284d319061baff17009a57 2023-03-29 03:55:25,803 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/A/dd266c63ee75421183eefedfab4f43c9 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/dd266c63ee75421183eefedfab4f43c9 2023-03-29 03:55:25,806 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/dd266c63ee75421183eefedfab4f43c9, entries=250, sequenceid=150, filesize=38.9 K 2023-03-29 03:55:25,807 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/B/2314c1fead8842ccb2a38abfbd527a3d as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/2314c1fead8842ccb2a38abfbd527a3d 2023-03-29 03:55:25,810 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/2314c1fead8842ccb2a38abfbd527a3d, entries=150, sequenceid=150, filesize=11.8 K 2023-03-29 03:55:25,810 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/C/45dba969a2284d319061baff17009a57 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/45dba969a2284d319061baff17009a57 2023-03-29 03:55:25,813 DEBUG [Thread-2498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:25,813 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/45dba969a2284d319061baff17009a57, entries=150, sequenceid=150, filesize=11.8 K 2023-03-29 03:55:25,814 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~280.64 KB/287376, currentSize=80.51 KB/82440 for a5c78075cc0f4fd1864c2f0339ad3d67 in 870ms, sequenceid=150, compaction requested=true 2023-03-29 03:55:25,814 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a5c78075cc0f4fd1864c2f0339ad3d67: 2023-03-29 03:55:25,815 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:55:25,815 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-29 03:55:25,815 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-29 03:55:25,815 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:55:25,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:25,815 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:55:25,817 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 161662 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-29 03:55:25,817 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 69122 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-29 03:55:25,817 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1912): a5c78075cc0f4fd1864c2f0339ad3d67/A is initiating minor compaction (all files) 2023-03-29 03:55:25,817 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1912): a5c78075cc0f4fd1864c2f0339ad3d67/B is initiating minor compaction (all files) 2023-03-29 03:55:25,817 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a5c78075cc0f4fd1864c2f0339ad3d67/A in TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:25,817 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a5c78075cc0f4fd1864c2f0339ad3d67/B in TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:25,817 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/413b333ae3934688bf785f28dd43be25, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/99006d66cc3e432b8e2abc2436babbc6, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/cafee5e0813f4a2da3cd17f07119e801, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/6ee45ee7b9204e39933bc2b40c5a9b19, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/b815d2bcbd3a487ba420d75dddbe1c6d, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/dd266c63ee75421183eefedfab4f43c9] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp, totalSize=157.9 K 2023-03-29 03:55:25,817 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/c79a2874dd184bcba8eb67a074c63a88, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/15a8b139d38f457d9284e7e2e260d532, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/5052efffd2a84a8d9080bcba50d3a88b, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/13fdff8451574686ab05e509355583bf, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/d45741edc43e4f80a0e4ad2e592dfc6a, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/2314c1fead8842ccb2a38abfbd527a3d] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp, totalSize=67.5 K 2023-03-29 03:55:25,817 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:25,817 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 413b333ae3934688bf785f28dd43be25, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1680062118489 2023-03-29 03:55:25,817 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting c79a2874dd184bcba8eb67a074c63a88, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1680062118489 2023-03-29 03:55:25,818 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 99006d66cc3e432b8e2abc2436babbc6, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=66, earliestPutTs=1680062118545 2023-03-29 03:55:25,818 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 15a8b139d38f457d9284e7e2e260d532, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=66, earliestPutTs=1680062118545 2023-03-29 03:55:25,818 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting cafee5e0813f4a2da3cd17f07119e801, keycount=100, bloomtype=ROW, size=18.4 K, encoding=NONE, compression=NONE, seqNum=74, earliestPutTs=1680062120686 2023-03-29 03:55:25,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:25,818 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 5052efffd2a84a8d9080bcba50d3a88b, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=74, earliestPutTs=1680062120686 2023-03-29 03:55:25,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:25,818 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 6ee45ee7b9204e39933bc2b40c5a9b19, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=111, earliestPutTs=1680062121397 2023-03-29 03:55:25,818 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:25,818 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:25,818 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:25,818 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:25,819 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 13fdff8451574686ab05e509355583bf, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=111, earliestPutTs=1680062121397 2023-03-29 03:55:25,819 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting b815d2bcbd3a487ba420d75dddbe1c6d, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=119, earliestPutTs=1680062122704 2023-03-29 03:55:25,819 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:25,819 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting d45741edc43e4f80a0e4ad2e592dfc6a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=119, earliestPutTs=1680062122704 2023-03-29 03:55:25,819 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:25,819 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting dd266c63ee75421183eefedfab4f43c9, keycount=250, bloomtype=ROW, size=38.9 K, encoding=NONE, compression=NONE, seqNum=150, earliestPutTs=1680062124913 2023-03-29 03:55:25,819 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 2314c1fead8842ccb2a38abfbd527a3d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=150, earliestPutTs=1680062124930 2023-03-29 03:55:25,822 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:25,822 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:25,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:25,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:25,823 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:25,823 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:25,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:25,823 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:25,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:25,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:25,824 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:25,825 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:25,825 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:25,825 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:25,825 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:25,825 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:25,826 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:25,826 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:25,826 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:25,826 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. started... 2023-03-29 03:55:25,827 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a5c78075cc0f4fd1864c2f0339ad3d67 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-03-29 03:55:25,827 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5c78075cc0f4fd1864c2f0339ad3d67, store=A 2023-03-29 03:55:25,827 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:55:25,827 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5c78075cc0f4fd1864c2f0339ad3d67, store=B 2023-03-29 03:55:25,827 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:55:25,827 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5c78075cc0f4fd1864c2f0339ad3d67, store=C 2023-03-29 03:55:25,828 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:55:25,834 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] throttle.PressureAwareThroughputController(145): a5c78075cc0f4fd1864c2f0339ad3d67#B#compaction#359 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:55:25,851 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] throttle.PressureAwareThroughputController(145): a5c78075cc0f4fd1864c2f0339ad3d67#A#compaction#358 average throughput is 1.04 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:55:25,855 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/B/f18327be7b684c08af91125d8c768797 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/f18327be7b684c08af91125d8c768797 2023-03-29 03:55:25,859 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in a5c78075cc0f4fd1864c2f0339ad3d67/B of a5c78075cc0f4fd1864c2f0339ad3d67 into f18327be7b684c08af91125d8c768797(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:55:25,859 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a5c78075cc0f4fd1864c2f0339ad3d67: 2023-03-29 03:55:25,859 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67., storeName=a5c78075cc0f4fd1864c2f0339ad3d67/B, priority=10, startTime=1680062125815; duration=0sec 2023-03-29 03:55:25,859 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:55:25,859 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-29 03:55:25,860 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 69122 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-29 03:55:25,860 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1912): a5c78075cc0f4fd1864c2f0339ad3d67/C is initiating minor compaction (all files) 2023-03-29 03:55:25,860 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a5c78075cc0f4fd1864c2f0339ad3d67/C in TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:25,861 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/7522e7a58f7749b4a733fc2e1bae07af, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/528af543643c4eed9cf12d5c2aa68c5e, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/dc646a8263e54b96b5315974d304b5af, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/eda0e120d6fa4644820b8f8b43605329, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/ebbc1e200ac34ceba35bf0c8c97c8c8e, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/45dba969a2284d319061baff17009a57] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp, totalSize=67.5 K 2023-03-29 03:55:25,861 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 7522e7a58f7749b4a733fc2e1bae07af, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1680062118489 2023-03-29 03:55:25,861 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 528af543643c4eed9cf12d5c2aa68c5e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=66, earliestPutTs=1680062118545 2023-03-29 03:55:25,861 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting dc646a8263e54b96b5315974d304b5af, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=74, earliestPutTs=1680062120686 2023-03-29 03:55:25,862 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting eda0e120d6fa4644820b8f8b43605329, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=111, earliestPutTs=1680062121397 2023-03-29 03:55:25,862 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting ebbc1e200ac34ceba35bf0c8c97c8c8e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=119, earliestPutTs=1680062122704 2023-03-29 03:55:25,862 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 45dba969a2284d319061baff17009a57, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=150, earliestPutTs=1680062124930 2023-03-29 03:55:25,871 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] throttle.PressureAwareThroughputController(145): a5c78075cc0f4fd1864c2f0339ad3d67#C#compaction#361 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:55:26,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on a5c78075cc0f4fd1864c2f0339ad3d67 2023-03-29 03:55:26,159 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. as already flushing 2023-03-29 03:55:26,208 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:26,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51990 deadline: 1680062186198, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:26,209 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:26,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51970 deadline: 1680062186199, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:26,272 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303298e9008f20e5940f5b89e2bf5752624c0 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303298e9008f20e5940f5b89e2bf5752624c0 2023-03-29 03:55:26,280 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=165, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/A/5cccd1ba78084bb8bbb93b502fa577bd 2023-03-29 03:55:26,287 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/C/e25bcd1d2d7b47a7a2a6c9a3172fe613 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/e25bcd1d2d7b47a7a2a6c9a3172fe613 2023-03-29 03:55:26,288 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/A/262d9d63ebaa47208cfe6461c39ece00 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/262d9d63ebaa47208cfe6461c39ece00 2023-03-29 03:55:26,293 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in a5c78075cc0f4fd1864c2f0339ad3d67/A of a5c78075cc0f4fd1864c2f0339ad3d67 into 262d9d63ebaa47208cfe6461c39ece00(size=25.5 K), total size for store is 25.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:55:26,294 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a5c78075cc0f4fd1864c2f0339ad3d67: 2023-03-29 03:55:26,294 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67., storeName=a5c78075cc0f4fd1864c2f0339ad3d67/A, priority=10, startTime=1680062125815; duration=0sec 2023-03-29 03:55:26,294 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:55:26,295 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in a5c78075cc0f4fd1864c2f0339ad3d67/C of a5c78075cc0f4fd1864c2f0339ad3d67 into e25bcd1d2d7b47a7a2a6c9a3172fe613(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:55:26,295 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a5c78075cc0f4fd1864c2f0339ad3d67: 2023-03-29 03:55:26,295 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67., storeName=a5c78075cc0f4fd1864c2f0339ad3d67/C, priority=10, startTime=1680062125815; duration=0sec 2023-03-29 03:55:26,295 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:55:26,311 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:26,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51990 deadline: 1680062186309, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:26,311 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:26,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51970 deadline: 1680062186310, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:26,516 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:26,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51990 deadline: 1680062186512, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:26,516 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:26,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51970 deadline: 1680062186512, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:26,702 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=165 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/B/7dff4f1e20944e93a93900eaae922eff 2023-03-29 03:55:26,712 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=165 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/C/88b32e3389054c6a9caf2e91ea917865 2023-03-29 03:55:26,715 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/A/5cccd1ba78084bb8bbb93b502fa577bd as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/5cccd1ba78084bb8bbb93b502fa577bd 2023-03-29 03:55:26,720 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/5cccd1ba78084bb8bbb93b502fa577bd, entries=150, sequenceid=165, filesize=25.3 K 2023-03-29 03:55:26,721 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/B/7dff4f1e20944e93a93900eaae922eff as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/7dff4f1e20944e93a93900eaae922eff 2023-03-29 03:55:26,724 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/7dff4f1e20944e93a93900eaae922eff, entries=150, sequenceid=165, filesize=11.8 K 2023-03-29 03:55:26,725 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/C/88b32e3389054c6a9caf2e91ea917865 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/88b32e3389054c6a9caf2e91ea917865 2023-03-29 03:55:26,730 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/88b32e3389054c6a9caf2e91ea917865, entries=150, sequenceid=165, filesize=11.8 K 2023-03-29 03:55:26,731 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for a5c78075cc0f4fd1864c2f0339ad3d67 in 904ms, sequenceid=165, compaction requested=false 2023-03-29 03:55:26,731 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a5c78075cc0f4fd1864c2f0339ad3d67: 2023-03-29 03:55:26,732 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:26,732 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:26,732 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:26,732 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:26,732 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:26,732 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:26,736 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:26,736 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:26,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:26,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:26,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:26,737 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:26,737 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:26,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:26,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:26,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:26,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:26,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:26,740 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:26,740 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5168a4bd[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:26,740 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:26,740 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:26,741 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:26,741 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:26,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:26,741 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:26,741 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:26,741 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:26,742 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:26,741 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:26,743 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:26,743 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:26,743 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:26,743 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:26,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:26,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:26,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:26,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:26,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:26,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:26,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:26,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:26,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:26,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:26,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:26,746 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:26,746 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:26,747 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:26,747 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:26,748 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:26,748 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:26,750 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:26,750 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:26,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:26,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:26,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:26,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:26,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:26,750 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:26,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:26,750 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:26,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:26,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:26,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:26,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:26,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:26,752 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:26,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:26,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:26,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:26,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:26,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:26,755 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:26,755 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:26,755 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:26,755 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:26,755 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:26,755 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:26,755 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:26,756 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:26,756 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:26,756 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:26,755 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:26,755 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:26,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:26,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:26,756 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:55:26,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:26,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:26,756 DEBUG [Thread-2498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:26,756 DEBUG [Thread-2498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:26,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on a5c78075cc0f4fd1864c2f0339ad3d67 2023-03-29 03:55:26,820 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a5c78075cc0f4fd1864c2f0339ad3d67 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-03-29 03:55:26,820 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5c78075cc0f4fd1864c2f0339ad3d67, store=A 2023-03-29 03:55:26,820 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:55:26,820 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5c78075cc0f4fd1864c2f0339ad3d67, store=B 2023-03-29 03:55:26,820 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:55:26,820 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5c78075cc0f4fd1864c2f0339ad3d67, store=C 2023-03-29 03:55:26,820 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:55:26,831 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230329d46615661c264b1e80b4113fe1427c1d to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230329d46615661c264b1e80b4113fe1427c1d 2023-03-29 03:55:26,848 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:26,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51990 deadline: 1680062186844, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:26,849 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:26,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51970 deadline: 1680062186845, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:26,857 DEBUG [Thread-2498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:26,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:26,858 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:26,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:26,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:26,859 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:26,859 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:26,860 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:26,860 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:26,860 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:26,860 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:26,861 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:26,861 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:26,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:26,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:26,861 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:26,861 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:26,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:26,862 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:26,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:26,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:26,862 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:26,862 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:26,862 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:26,862 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:26,863 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:26,863 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:26,863 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:26,863 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:26,863 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:26,863 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. started... 2023-03-29 03:55:26,863 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. as already flushing 2023-03-29 03:55:26,863 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:26,863 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:26,863 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:26,863 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:26,863 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:26,863 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:26,865 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:26,865 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:26,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:26,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:26,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:26,865 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:26,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:26,865 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:26,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:26,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:26,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:26,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:26,866 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:26,866 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f2ab888[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:26,866 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:26,867 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:26,868 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:26,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:26,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:26,868 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:26,868 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:26,868 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:26,868 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:26,868 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:26,871 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:26,871 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:26,871 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:26,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:26,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:26,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:26,871 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:26,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:26,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:26,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:26,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:26,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:26,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:26,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:26,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:26,873 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:26,873 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:26,873 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:26,873 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:26,873 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:26,873 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:26,875 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:26,875 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:26,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:26,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:26,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:26,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:26,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:26,875 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:26,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:26,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:26,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:26,876 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:26,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:26,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:26,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:26,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:26,877 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:26,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:26,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:26,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:26,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:26,879 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:26,879 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:26,879 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:26,879 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:26,879 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:26,879 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:26,879 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:26,879 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:26,879 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:26,879 DEBUG [Thread-2498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:26,879 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:55:26,879 DEBUG [Thread-2498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:26,879 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:26,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:26,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:26,879 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:26,880 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:26,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:26,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:26,952 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:26,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51990 deadline: 1680062186949, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:26,954 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:26,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51970 deadline: 1680062186950, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:26,980 DEBUG [Thread-2498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:26,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:26,981 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:26,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:26,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:26,982 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:26,983 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:26,983 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:26,983 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:26,983 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:26,983 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:26,984 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:26,984 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:26,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:26,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:26,985 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:26,985 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:26,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:26,985 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:26,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:26,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:26,985 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:26,985 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:26,985 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:26,986 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:26,986 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:26,986 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:26,986 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:26,986 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:26,986 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:26,986 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. started... 2023-03-29 03:55:26,986 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. as already flushing 2023-03-29 03:55:26,986 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:26,986 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:26,986 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:26,986 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:26,986 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:26,986 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:26,988 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:26,988 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:26,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:26,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:26,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:26,989 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:26,989 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:26,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:26,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:26,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:26,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:26,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:26,990 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:26,990 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3a028c01[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:26,990 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:26,990 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:26,991 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:26,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:26,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:26,991 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:26,991 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:26,991 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:26,991 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:26,991 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:26,993 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:26,993 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:26,993 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:26,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:26,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:26,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:26,993 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:26,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:26,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:26,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:26,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:26,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:26,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:26,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:26,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:26,996 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:26,996 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:26,997 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:26,997 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:26,997 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:26,997 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:26,998 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:26,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:26,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:26,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:26,998 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:26,998 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:26,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:26,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:26,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:26,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:26,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:27,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:27,000 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:27,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:27,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:27,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:27,001 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:27,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:27,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:27,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:27,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:27,004 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:27,004 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:27,004 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:27,004 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:27,004 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:27,004 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:27,005 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:27,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:27,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:27,004 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:27,005 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:55:27,005 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:27,005 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:27,005 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:27,005 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:27,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:27,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:27,005 DEBUG [Thread-2498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:27,005 DEBUG [Thread-2498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:27,105 DEBUG [Thread-2498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:27,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:27,107 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:27,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:27,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:27,109 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:27,109 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:27,109 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:27,109 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:27,109 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:27,109 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:27,110 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:27,110 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:27,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:27,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:27,111 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:27,111 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:27,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:27,111 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:27,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:27,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:27,112 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:27,112 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:27,112 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:27,112 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:27,112 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:27,112 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:27,112 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:27,112 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:27,112 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:27,112 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. started... 2023-03-29 03:55:27,112 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. as already flushing 2023-03-29 03:55:27,112 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:27,113 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:27,113 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:27,113 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:27,113 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:27,113 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:27,116 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:27,116 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:27,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:27,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:27,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:27,117 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:27,117 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:27,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:27,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:27,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:27,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:27,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:27,118 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:27,118 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@72ebabef[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:27,118 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:27,118 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:27,120 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:27,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:27,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:27,120 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:27,120 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:27,120 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:27,120 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:27,120 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:27,122 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:27,122 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:27,122 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:27,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:27,122 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:27,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:27,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:27,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:27,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:27,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:27,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:27,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:27,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:27,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:27,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:27,124 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:27,124 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:27,124 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:27,124 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:27,124 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:27,124 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:27,126 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:27,126 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:27,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:27,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:27,126 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:27,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:27,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:27,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:27,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:27,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:27,126 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:27,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:27,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:27,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:27,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:27,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:27,128 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:27,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:27,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:27,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:27,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:27,130 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:27,130 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:27,130 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:27,130 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:27,130 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:27,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:27,130 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:27,130 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:27,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:27,130 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:27,130 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:27,130 DEBUG [Thread-2498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:27,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:27,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:27,130 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:27,131 DEBUG [Thread-2498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:27,131 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:27,131 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:27,156 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:27,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51990 deadline: 1680062187153, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:27,156 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:27,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51970 deadline: 1680062187155, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:27,231 DEBUG [Thread-2498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:27,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:27,234 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:27,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:27,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:27,235 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:27,235 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:27,235 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:27,235 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:27,236 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:27,236 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:27,237 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=186, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/A/217fd2cf2cf44201b12572d4b9218c9d 2023-03-29 03:55:27,238 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:27,238 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:27,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:27,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:27,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:27,239 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:27,240 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:27,240 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:27,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:27,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:27,240 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:27,240 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:27,240 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:27,240 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:27,240 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:27,240 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:27,242 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:27,242 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:27,242 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:27,242 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. started... 2023-03-29 03:55:27,242 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. as already flushing 2023-03-29 03:55:27,242 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:27,242 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:27,242 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:27,242 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:27,242 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:27,242 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:27,244 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:27,244 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:27,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:27,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:27,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:27,244 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:27,244 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:27,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:27,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:27,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:27,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:27,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:27,247 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:27,247 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6632a443[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:27,247 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:27,247 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:27,248 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:27,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:27,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:27,248 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:27,248 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:27,248 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:27,249 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:27,248 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:27,250 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:27,250 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:27,250 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:27,250 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:27,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:27,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:27,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:27,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:27,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:27,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:27,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:27,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:27,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:27,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:27,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:27,253 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:27,253 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:27,254 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:27,254 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:27,254 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:27,254 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:27,258 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:27,258 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:27,258 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:27,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:27,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:27,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:27,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:27,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:27,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:27,259 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:27,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:27,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:27,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:27,261 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:27,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:27,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:27,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:27,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:27,264 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:27,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:27,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:27,264 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:27,265 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:27,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:27,265 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:27,265 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:27,265 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:27,265 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:27,265 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:27,265 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:27,265 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:27,265 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:27,265 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:27,265 DEBUG [Thread-2498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:27,265 DEBUG [Thread-2498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:27,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:27,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:27,273 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:55:27,365 DEBUG [Thread-2498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:27,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:27,368 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:27,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:27,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:27,372 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:27,372 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:27,372 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:27,372 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:27,373 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:27,373 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:27,374 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:27,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:27,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:27,374 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:27,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:27,374 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:27,375 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:27,375 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:27,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:27,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:27,375 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:27,376 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:27,376 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:27,376 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:27,376 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:27,376 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:27,380 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:27,380 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:27,380 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:27,380 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. started... 2023-03-29 03:55:27,380 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. as already flushing 2023-03-29 03:55:27,380 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:27,380 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:27,380 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:27,380 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:27,380 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:27,380 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:27,383 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:27,383 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:27,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:27,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:27,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:27,383 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:27,383 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:27,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:27,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:27,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:27,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:27,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:27,387 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:27,387 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ce6aef5[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:27,387 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:27,387 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:27,388 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:27,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:27,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:27,388 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:27,388 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:27,389 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:27,389 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:27,389 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:27,391 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:27,392 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:27,391 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:27,392 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:27,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:27,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:27,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:27,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:27,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:27,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:27,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:27,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:27,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:27,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:27,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:27,394 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:27,394 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:27,395 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:27,395 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:27,395 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:27,395 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:27,397 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:27,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:27,397 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:27,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:27,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:27,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:27,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:27,397 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:27,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:27,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:27,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:27,399 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:27,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:27,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:27,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:27,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:27,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:27,401 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:27,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:27,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:27,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:27,409 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:27,409 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:27,409 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:27,409 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:27,409 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:27,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:27,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:27,409 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:27,409 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:27,409 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:27,409 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:27,409 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:27,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:27,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:27,409 DEBUG [Thread-2498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:27,409 DEBUG [Thread-2498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:27,409 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:27,409 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:27,463 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:27,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51990 deadline: 1680062187458, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:27,467 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:27,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51970 deadline: 1680062187459, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:27,510 DEBUG [Thread-2498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:27,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:27,512 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:27,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:27,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:27,513 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:27,513 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:27,513 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:27,513 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:27,513 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:27,513 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:27,515 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:27,515 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:27,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:27,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:27,515 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:27,515 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:27,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:27,516 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:27,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:27,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:27,516 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:27,516 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:27,516 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:27,516 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:27,516 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:27,516 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:27,517 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:27,517 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:27,517 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:27,517 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. started... 2023-03-29 03:55:27,517 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. as already flushing 2023-03-29 03:55:27,517 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:27,517 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:27,517 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:27,517 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:27,517 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:27,517 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:27,518 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:27,518 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:27,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:27,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:27,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:27,519 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:27,519 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:27,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:27,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:27,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:27,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:27,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:27,521 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:27,521 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@722a53ec[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:27,521 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:27,521 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:27,522 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:27,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:27,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:27,522 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:27,522 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:27,522 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:27,522 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:27,522 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:27,526 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:27,526 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:27,527 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:27,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:27,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:27,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:27,527 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:27,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:27,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:27,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:27,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:27,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:27,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:27,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:27,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:27,529 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:27,529 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:27,529 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:27,529 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:27,529 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:27,529 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:27,531 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:27,531 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:27,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:27,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:27,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:27,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:27,531 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:27,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:27,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:27,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:27,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:27,532 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:27,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:27,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:27,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:27,533 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:27,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:27,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:27,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:27,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:27,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:27,535 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:27,535 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:27,535 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:27,536 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:27,535 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:27,536 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:27,536 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:27,536 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:27,536 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:27,536 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:27,536 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:27,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:27,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:27,536 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:27,536 DEBUG [Thread-2498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:27,536 DEBUG [Thread-2498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:27,536 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:55:27,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:27,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:27,636 DEBUG [Thread-2498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:27,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:27,639 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:27,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:27,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:27,640 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:27,640 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:27,640 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:27,640 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:27,642 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:27,642 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:27,643 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:27,643 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:27,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:27,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:27,644 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:27,644 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:27,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:27,644 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:27,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:27,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:27,645 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:27,645 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:27,645 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:27,645 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:27,645 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:27,645 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:27,645 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:27,645 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:27,645 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:27,645 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. started... 2023-03-29 03:55:27,645 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. as already flushing 2023-03-29 03:55:27,645 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:27,646 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:27,646 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:27,646 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:27,646 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:27,646 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:27,648 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:27,648 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:27,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:27,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:27,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:27,649 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:27,649 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:27,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:27,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:27,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:27,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:27,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:27,650 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:27,650 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5db13a95[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:27,650 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:27,650 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:27,651 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:27,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:27,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:27,652 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:27,652 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:27,652 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:27,652 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:27,652 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:27,653 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:27,653 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:27,653 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:27,653 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:27,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:27,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:27,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:27,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:27,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:27,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:27,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:27,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:27,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:27,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:27,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:27,656 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:27,656 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:27,656 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:27,656 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:27,656 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:27,656 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:27,659 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:27,659 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:27,659 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:27,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:27,659 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:27,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:27,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:27,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:27,659 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:27,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:27,660 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:27,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:27,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:27,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:27,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:27,661 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:27,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:27,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:27,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:27,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:27,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:27,664 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:27,664 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:27,664 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:27,664 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:27,664 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:27,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:27,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:27,664 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:27,664 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:27,664 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:27,664 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:27,664 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:27,664 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:27,664 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:27,664 DEBUG [Thread-2498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:27,664 DEBUG [Thread-2498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:27,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:27,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:27,665 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:55:27,674 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=186 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/B/d1a227f3118c40498b880f98c929c7b7 2023-03-29 03:55:27,765 DEBUG [Thread-2498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:27,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:27,768 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:27,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:27,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:27,774 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:27,774 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:27,774 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:27,774 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:27,774 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:27,774 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:27,776 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:27,776 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:27,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:27,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:27,776 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:27,776 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:27,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:27,776 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:27,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:27,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:27,777 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:27,777 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:27,777 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:27,777 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:27,777 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:27,777 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:27,777 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:27,777 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:27,777 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:27,777 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. started... 2023-03-29 03:55:27,777 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. as already flushing 2023-03-29 03:55:27,777 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:27,777 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:27,778 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:27,778 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:27,778 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:27,778 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:27,779 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:27,779 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:27,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:27,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:27,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:27,780 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:27,780 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:27,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:27,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:27,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:27,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:27,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:27,781 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:27,782 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ed29ddc[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:27,782 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:27,782 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:27,783 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:27,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:27,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:27,783 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:27,783 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:27,783 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:27,783 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:27,783 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:27,786 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:27,786 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:27,786 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:27,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:27,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:27,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:27,786 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:27,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:27,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:27,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:27,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:27,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:27,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:27,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:27,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:27,788 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:27,788 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:27,788 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:27,788 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:27,788 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:27,788 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:27,790 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:27,790 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:27,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:27,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:27,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:27,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:27,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:27,790 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:27,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:27,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:27,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:27,791 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:27,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:27,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:27,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:27,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:27,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:27,792 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:27,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:27,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:27,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:27,794 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:27,794 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:27,794 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:27,794 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:27,794 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:27,794 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:27,794 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:27,794 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:27,794 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:27,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:27,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:27,795 DEBUG [Thread-2498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:27,794 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:27,795 DEBUG [Thread-2498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:27,795 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:27,795 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:27,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:27,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:27,895 DEBUG [Thread-2498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:27,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:27,897 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:27,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:27,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:27,898 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:27,898 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:27,898 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:27,898 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:27,899 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:27,899 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:27,900 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:27,900 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:27,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:27,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:27,900 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:27,900 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:27,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:27,901 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:27,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:27,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:27,901 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:27,901 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:27,901 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:27,901 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:27,901 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:27,901 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:27,902 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:27,902 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:27,902 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:27,902 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. started... 2023-03-29 03:55:27,902 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. as already flushing 2023-03-29 03:55:27,902 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:27,902 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:27,902 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:27,902 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:27,902 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:27,902 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:27,905 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:27,905 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:27,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:27,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:27,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:27,905 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:27,905 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:27,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:27,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:27,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:27,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:27,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:27,906 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:27,906 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@47aa0ff5[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:27,906 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:27,907 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:27,908 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:27,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:27,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:27,908 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:27,908 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:27,908 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:27,908 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:27,908 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:27,909 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:27,909 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:27,909 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:27,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:27,909 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:27,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:27,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:27,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:27,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:27,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:27,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:27,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:27,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:27,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:27,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:27,912 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:27,912 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:27,912 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:27,912 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:27,912 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:27,912 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:27,913 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:27,913 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:27,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:27,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:27,913 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:27,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:27,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:27,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:27,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:27,914 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:27,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:27,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:27,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:27,915 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:27,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:27,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:27,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:27,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:27,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:27,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:27,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:27,921 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:27,921 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:27,921 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:27,921 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:55:27,921 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:27,921 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:55:27,921 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:27,921 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:27,921 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:27,921 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:27,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:27,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:27,921 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:27,922 DEBUG [Thread-2498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:27,922 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:27,922 DEBUG [Thread-2498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:27,922 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:27,922 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:27,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:27,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:27,968 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:27,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51990 deadline: 1680062187964, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:27,970 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:27,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51970 deadline: 1680062187969, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:28,022 DEBUG [Thread-2498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:28,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:28,024 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:28,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:28,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:28,025 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:28,025 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:28,025 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:28,025 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:28,025 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:28,025 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:28,027 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:28,027 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:28,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:28,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:28,027 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:28,027 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:28,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:28,027 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:28,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:28,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:28,028 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:28,028 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:28,028 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:28,028 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:28,028 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:28,028 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:28,028 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:28,029 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:28,029 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:28,029 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. started... 2023-03-29 03:55:28,029 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. as already flushing 2023-03-29 03:55:28,029 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:28,029 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:28,029 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:28,029 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:28,029 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:28,029 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:28,030 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:28,030 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:28,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:28,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:28,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:28,031 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:28,031 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:28,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:28,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:28,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:28,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:28,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:28,032 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:28,032 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:28,032 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@673f91df[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:28,032 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:28,033 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:28,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:28,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:28,034 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:28,034 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:28,034 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:28,034 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:28,034 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:28,036 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:28,036 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:28,036 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:28,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:28,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:28,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:28,036 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:28,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:28,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:28,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:28,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:28,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:28,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:28,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:28,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:28,038 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:28,038 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:28,039 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:28,039 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:28,039 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:28,039 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:28,040 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:28,040 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:28,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:28,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:28,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:28,040 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:28,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:28,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:28,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:28,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:28,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:28,041 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:28,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:28,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:28,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:28,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:28,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:28,042 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:28,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:28,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:28,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:28,044 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:28,044 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:28,044 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:28,044 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:28,044 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:28,044 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:28,044 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:28,044 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:28,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:28,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:28,044 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:28,044 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:28,045 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:28,044 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:28,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:28,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:28,045 DEBUG [Thread-2498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:28,045 DEBUG [Thread-2498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:28,085 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=186 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/C/002ddcbbb4c84fe58514457890b34e3d 2023-03-29 03:55:28,089 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/A/217fd2cf2cf44201b12572d4b9218c9d as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/217fd2cf2cf44201b12572d4b9218c9d 2023-03-29 03:55:28,092 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/217fd2cf2cf44201b12572d4b9218c9d, entries=200, sequenceid=186, filesize=32.1 K 2023-03-29 03:55:28,093 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/B/d1a227f3118c40498b880f98c929c7b7 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/d1a227f3118c40498b880f98c929c7b7 2023-03-29 03:55:28,096 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/d1a227f3118c40498b880f98c929c7b7, entries=150, sequenceid=186, filesize=11.8 K 2023-03-29 03:55:28,097 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/C/002ddcbbb4c84fe58514457890b34e3d as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/002ddcbbb4c84fe58514457890b34e3d 2023-03-29 03:55:28,100 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/002ddcbbb4c84fe58514457890b34e3d, entries=150, sequenceid=186, filesize=11.8 K 2023-03-29 03:55:28,101 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for a5c78075cc0f4fd1864c2f0339ad3d67 in 1281ms, sequenceid=186, compaction requested=true 2023-03-29 03:55:28,101 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a5c78075cc0f4fd1864c2f0339ad3d67: 2023-03-29 03:55:28,101 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:55:28,101 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-29 03:55:28,101 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-29 03:55:28,101 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:55:28,101 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:55:28,102 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 84884 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-29 03:55:28,102 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36314 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-29 03:55:28,102 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1912): a5c78075cc0f4fd1864c2f0339ad3d67/A is initiating minor compaction (all files) 2023-03-29 03:55:28,102 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1912): a5c78075cc0f4fd1864c2f0339ad3d67/B is initiating minor compaction (all files) 2023-03-29 03:55:28,102 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a5c78075cc0f4fd1864c2f0339ad3d67/A in TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:28,102 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a5c78075cc0f4fd1864c2f0339ad3d67/B in TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:28,102 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/262d9d63ebaa47208cfe6461c39ece00, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/5cccd1ba78084bb8bbb93b502fa577bd, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/217fd2cf2cf44201b12572d4b9218c9d] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp, totalSize=82.9 K 2023-03-29 03:55:28,103 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/f18327be7b684c08af91125d8c768797, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/7dff4f1e20944e93a93900eaae922eff, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/d1a227f3118c40498b880f98c929c7b7] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp, totalSize=35.5 K 2023-03-29 03:55:28,103 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 262d9d63ebaa47208cfe6461c39ece00, keycount=150, bloomtype=ROW, size=25.5 K, encoding=NONE, compression=NONE, seqNum=150, earliestPutTs=1680062124930 2023-03-29 03:55:28,103 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting f18327be7b684c08af91125d8c768797, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=150, earliestPutTs=1680062124930 2023-03-29 03:55:28,103 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 5cccd1ba78084bb8bbb93b502fa577bd, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=165, earliestPutTs=1680062124977 2023-03-29 03:55:28,103 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 7dff4f1e20944e93a93900eaae922eff, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=165, earliestPutTs=1680062124977 2023-03-29 03:55:28,104 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 217fd2cf2cf44201b12572d4b9218c9d, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=186, earliestPutTs=1680062126190 2023-03-29 03:55:28,104 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting d1a227f3118c40498b880f98c929c7b7, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=186, earliestPutTs=1680062126190 2023-03-29 03:55:28,110 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] throttle.PressureAwareThroughputController(145): a5c78075cc0f4fd1864c2f0339ad3d67#B#compaction#368 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:55:28,113 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] throttle.PressureAwareThroughputController(145): a5c78075cc0f4fd1864c2f0339ad3d67#A#compaction#367 average throughput is 4.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:55:28,129 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/B/449af9b48e734a02b5e692557b1a1e17 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/449af9b48e734a02b5e692557b1a1e17 2023-03-29 03:55:28,142 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a5c78075cc0f4fd1864c2f0339ad3d67/B of a5c78075cc0f4fd1864c2f0339ad3d67 into 449af9b48e734a02b5e692557b1a1e17(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:55:28,142 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a5c78075cc0f4fd1864c2f0339ad3d67: 2023-03-29 03:55:28,142 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67., storeName=a5c78075cc0f4fd1864c2f0339ad3d67/B, priority=13, startTime=1680062128101; duration=0sec 2023-03-29 03:55:28,143 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:55:28,143 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-29 03:55:28,144 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36314 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-29 03:55:28,144 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1912): a5c78075cc0f4fd1864c2f0339ad3d67/C is initiating minor compaction (all files) 2023-03-29 03:55:28,144 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a5c78075cc0f4fd1864c2f0339ad3d67/C in TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:28,145 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/e25bcd1d2d7b47a7a2a6c9a3172fe613, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/88b32e3389054c6a9caf2e91ea917865, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/002ddcbbb4c84fe58514457890b34e3d] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp, totalSize=35.5 K 2023-03-29 03:55:28,145 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting e25bcd1d2d7b47a7a2a6c9a3172fe613, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=150, earliestPutTs=1680062124930 2023-03-29 03:55:28,145 DEBUG [Thread-2498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:28,145 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 88b32e3389054c6a9caf2e91ea917865, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=165, earliestPutTs=1680062124977 2023-03-29 03:55:28,146 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 002ddcbbb4c84fe58514457890b34e3d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=186, earliestPutTs=1680062126190 2023-03-29 03:55:28,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:28,152 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] throttle.PressureAwareThroughputController(145): a5c78075cc0f4fd1864c2f0339ad3d67#C#compaction#369 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:55:28,154 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:28,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:28,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:28,155 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:28,155 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:28,155 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:28,155 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:28,155 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:28,155 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:28,156 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:28,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:28,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:28,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:28,157 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:28,157 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:28,158 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:28,158 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:28,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:28,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:28,158 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:28,169 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:28,169 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:28,169 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:28,169 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:28,170 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:28,170 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:28,170 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:28,170 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. started... 2023-03-29 03:55:28,170 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a5c78075cc0f4fd1864c2f0339ad3d67 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-03-29 03:55:28,170 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5c78075cc0f4fd1864c2f0339ad3d67, store=A 2023-03-29 03:55:28,170 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:55:28,170 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5c78075cc0f4fd1864c2f0339ad3d67, store=B 2023-03-29 03:55:28,170 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:55:28,170 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5c78075cc0f4fd1864c2f0339ad3d67, store=C 2023-03-29 03:55:28,171 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:55:28,175 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/C/5e7b29bba65940acabd3c8d28c38897d as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/5e7b29bba65940acabd3c8d28c38897d 2023-03-29 03:55:28,180 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a5c78075cc0f4fd1864c2f0339ad3d67/C of a5c78075cc0f4fd1864c2f0339ad3d67 into 5e7b29bba65940acabd3c8d28c38897d(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:55:28,180 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a5c78075cc0f4fd1864c2f0339ad3d67: 2023-03-29 03:55:28,180 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67., storeName=a5c78075cc0f4fd1864c2f0339ad3d67/C, priority=13, startTime=1680062128101; duration=0sec 2023-03-29 03:55:28,181 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:55:28,193 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032988e1d2764d8a4466965d4a7f5df539ef to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032988e1d2764d8a4466965d4a7f5df539ef 2023-03-29 03:55:28,199 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=201, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/A/8848dbbad301430ea69d561a16c84f9f 2023-03-29 03:55:28,212 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=201 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/B/a2e31e392f844f6698bae1872e0f9b15 2023-03-29 03:55:28,239 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=201 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/C/d0ed71e2627849f195cefc1eefb1ff72 2023-03-29 03:55:28,251 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/A/8848dbbad301430ea69d561a16c84f9f as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/8848dbbad301430ea69d561a16c84f9f 2023-03-29 03:55:28,256 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/8848dbbad301430ea69d561a16c84f9f, entries=150, sequenceid=201, filesize=25.3 K 2023-03-29 03:55:28,257 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/B/a2e31e392f844f6698bae1872e0f9b15 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/a2e31e392f844f6698bae1872e0f9b15 2023-03-29 03:55:28,266 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/a2e31e392f844f6698bae1872e0f9b15, entries=150, sequenceid=201, filesize=11.8 K 2023-03-29 03:55:28,266 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/C/d0ed71e2627849f195cefc1eefb1ff72 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/d0ed71e2627849f195cefc1eefb1ff72 2023-03-29 03:55:28,275 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/d0ed71e2627849f195cefc1eefb1ff72, entries=150, sequenceid=201, filesize=11.8 K 2023-03-29 03:55:28,276 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=0 B/0 for a5c78075cc0f4fd1864c2f0339ad3d67 in 106ms, sequenceid=201, compaction requested=false 2023-03-29 03:55:28,276 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a5c78075cc0f4fd1864c2f0339ad3d67: 2023-03-29 03:55:28,276 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:28,276 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:28,276 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:28,276 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:28,276 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:28,276 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:28,279 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:28,279 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:28,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:28,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:28,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:28,279 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:28,279 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:28,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:28,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:28,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:28,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:28,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:28,282 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:28,282 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@338e7a3f[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:28,282 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:28,282 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:28,284 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:28,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:28,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:28,284 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:28,284 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:28,284 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:28,284 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:28,284 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:28,286 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:28,286 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:28,286 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:28,286 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:28,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:28,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:28,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:28,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:28,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:28,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:28,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:28,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:28,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:28,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:28,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:28,289 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:28,289 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:28,289 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:28,289 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:28,289 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:28,289 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:28,291 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:28,291 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:28,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:28,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:28,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:28,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:28,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:28,292 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:28,293 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:28,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:28,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:28,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:28,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:28,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:28,295 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:28,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:28,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:28,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:28,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:28,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:28,299 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:28,299 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:28,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:28,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:28,299 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:28,299 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:28,299 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:28,299 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:28,299 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:28,299 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:28,299 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:28,299 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:28,299 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:28,299 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:28,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:28,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:28,300 DEBUG [Thread-2498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:28,300 DEBUG [Thread-2498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:28,300 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:55:28,400 DEBUG [Thread-2498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:28,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:28,402 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:28,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:28,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:28,404 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:28,404 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:28,404 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:28,404 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:28,404 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:28,404 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:28,405 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:28,405 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:28,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:28,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:28,406 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:28,406 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:28,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:28,406 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:28,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:28,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:28,407 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:28,407 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:28,407 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:28,407 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:28,407 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:28,407 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:28,407 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:28,407 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:28,407 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:28,407 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. started... 2023-03-29 03:55:28,407 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a5c78075cc0f4fd1864c2f0339ad3d67: 2023-03-29 03:55:28,407 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:28,407 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:28,407 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:28,407 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:28,408 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:28,408 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:28,409 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:28,409 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:28,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:28,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:28,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:28,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:28,410 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:28,410 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:28,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:28,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:28,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:28,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:28,411 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:28,411 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@9083def[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:28,411 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:28,411 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:28,413 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:28,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:28,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:28,413 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:28,413 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:28,413 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:28,413 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:28,413 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:28,415 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:28,415 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:28,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:28,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:28,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:28,415 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:28,416 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:28,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:28,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:28,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:28,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:28,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:28,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:28,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:28,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:28,418 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:28,418 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:28,418 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:28,418 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:28,418 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:28,418 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:28,420 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:28,420 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:28,420 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:28,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:28,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:28,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:28,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:28,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:28,420 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:28,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:28,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:28,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:28,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:28,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:28,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:28,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:28,422 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:28,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:28,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:28,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:28,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:28,427 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:28,427 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:28,427 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:28,427 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:28,427 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:28,427 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:28,427 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:28,427 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:28,427 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:28,427 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:28,427 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:28,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:28,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:28,427 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:28,427 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:55:28,428 DEBUG [Thread-2498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:28,428 DEBUG [Thread-2498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:28,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:28,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:28,528 DEBUG [Thread-2498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:28,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:28,530 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:28,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:28,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:28,531 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:28,531 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:28,531 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:28,531 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:28,532 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:28,532 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:28,534 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:28,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:28,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:28,534 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:28,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:28,535 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:28,535 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:28,535 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:28,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:28,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:28,535 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:28,535 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:28,535 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:28,536 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:28,536 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:28,536 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:28,537 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:28,537 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:28,537 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:28,537 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. started... 2023-03-29 03:55:28,537 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a5c78075cc0f4fd1864c2f0339ad3d67: 2023-03-29 03:55:28,537 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:28,537 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:28,537 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:28,537 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:28,537 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:28,537 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:28,539 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:28,539 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:28,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:28,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:28,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:28,539 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:28,539 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:28,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:28,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:28,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:28,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:28,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:28,541 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:28,541 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e4a5e9c[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:28,541 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:28,541 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:28,542 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:28,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:28,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:28,542 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:28,542 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:28,542 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:28,542 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:28,542 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:28,544 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:28,544 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:28,544 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:28,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:28,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:28,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:28,544 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:28,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:28,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:28,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:28,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:28,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:28,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:28,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:28,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:28,547 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:28,547 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:28,547 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:28,547 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:28,547 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:28,547 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:28,553 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/A/496176f4abe44d389b39e070052d006f as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/496176f4abe44d389b39e070052d006f 2023-03-29 03:55:28,554 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:28,554 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:28,554 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:28,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:28,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:28,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:28,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:28,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:28,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:28,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:28,555 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:28,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:28,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:28,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:28,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:28,556 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:28,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:28,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:28,557 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a5c78075cc0f4fd1864c2f0339ad3d67/A of a5c78075cc0f4fd1864c2f0339ad3d67 into 496176f4abe44d389b39e070052d006f(size=25.6 K), total size for store is 50.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:55:28,557 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a5c78075cc0f4fd1864c2f0339ad3d67: 2023-03-29 03:55:28,557 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67., storeName=a5c78075cc0f4fd1864c2f0339ad3d67/A, priority=13, startTime=1680062128101; duration=0sec 2023-03-29 03:55:28,557 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:55:28,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:28,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:28,559 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:28,559 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:28,559 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:28,559 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:28,559 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:28,559 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:28,559 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:28,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:28,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:28,559 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:28,559 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:28,559 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:28,559 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:55:28,559 DEBUG [Thread-2498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:28,560 DEBUG [Thread-2498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:28,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:28,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:28,559 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:28,560 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:28,660 DEBUG [Thread-2498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:28,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:28,662 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:28,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:28,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:28,663 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:28,663 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:28,663 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:28,663 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:28,664 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:28,664 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:28,665 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:28,665 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:28,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:28,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:28,665 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:28,666 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:28,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:28,666 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:28,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:28,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:28,666 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:28,667 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:28,667 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:28,667 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:28,667 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:28,667 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:28,667 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:28,667 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:28,667 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:28,667 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. started... 2023-03-29 03:55:28,667 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a5c78075cc0f4fd1864c2f0339ad3d67: 2023-03-29 03:55:28,667 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:28,667 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:28,667 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:28,667 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:28,667 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:28,667 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:28,669 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:28,669 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:28,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:28,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:28,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:28,669 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:28,669 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:28,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:28,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:28,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:28,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:28,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:28,671 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:28,671 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e085e3[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:28,671 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:28,671 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:28,673 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:28,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:28,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:28,673 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:28,673 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:28,673 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:28,673 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:28,674 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:28,675 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:28,675 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:28,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:28,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:28,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:28,675 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:28,675 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:28,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:28,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:28,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:28,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:28,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:28,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:28,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:28,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:28,677 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:28,677 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:28,678 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:28,678 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:28,678 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:28,678 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:28,679 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:28,679 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:28,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:28,679 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:28,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:28,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:28,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:28,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:28,680 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:28,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:28,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:28,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:28,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:28,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:28,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:28,681 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:28,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:28,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:28,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:28,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:28,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:28,683 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:28,683 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:28,683 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:28,683 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:28,683 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:28,683 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:28,683 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:28,683 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:28,683 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:28,683 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:28,683 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:28,683 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:55:28,683 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:28,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:28,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:28,683 DEBUG [Thread-2498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:28,684 DEBUG [Thread-2498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:28,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:28,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:28,784 DEBUG [Thread-2498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:28,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:28,786 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:28,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:28,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:28,787 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:28,787 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:28,787 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:28,787 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:28,788 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:28,788 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:28,800 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:28,800 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:28,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:28,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:28,801 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:28,801 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:28,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:28,801 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:28,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:28,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:28,802 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:28,802 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:28,802 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:28,802 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:28,802 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:28,802 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:28,802 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:28,802 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:28,802 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:28,802 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. started... 2023-03-29 03:55:28,803 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a5c78075cc0f4fd1864c2f0339ad3d67: 2023-03-29 03:55:28,803 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:28,803 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:28,803 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:28,803 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:28,803 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:28,803 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:28,805 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:28,805 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:28,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:28,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:28,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:28,805 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:28,806 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:28,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:28,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:28,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:28,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:28,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:28,807 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:28,807 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@45ff58e2[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:28,807 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:28,807 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:28,808 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:28,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:28,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:28,808 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:28,808 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:28,808 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:28,808 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:28,808 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:28,810 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:28,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:28,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:28,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:28,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:28,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:28,811 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:28,811 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:28,811 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:28,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:28,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:28,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:28,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:28,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:28,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:28,813 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:28,813 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:28,813 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:28,813 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:28,813 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:28,813 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:28,815 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:28,815 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:28,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:28,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:28,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:28,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:28,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:28,815 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:28,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:28,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:28,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:28,816 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:28,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:28,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:28,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:28,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:28,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:28,817 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:28,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:28,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:28,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:28,820 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:28,820 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:28,820 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:28,820 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:28,820 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:28,820 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:28,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:28,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:28,820 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:28,820 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:28,820 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:28,820 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:28,820 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:28,820 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:28,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:28,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:28,820 DEBUG [Thread-2498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:28,821 DEBUG [Thread-2498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:28,921 DEBUG [Thread-2498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:28,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:28,923 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:28,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:28,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:28,924 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:28,924 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:28,924 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:28,924 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:28,925 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:28,925 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:28,929 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:28,929 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:28,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:28,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:28,930 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:28,930 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:28,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:28,930 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:28,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:28,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:28,930 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:28,931 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:28,931 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:28,931 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:28,931 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:28,931 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:28,931 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:28,931 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:28,931 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:28,931 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. started... 2023-03-29 03:55:28,931 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a5c78075cc0f4fd1864c2f0339ad3d67: 2023-03-29 03:55:28,931 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:28,932 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:28,933 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:28,933 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:28,933 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:28,933 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:28,935 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:28,935 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:28,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:28,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:28,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:28,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:28,936 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:28,936 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:28,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:28,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:28,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:28,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:28,937 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:28,937 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@dd0f2f7[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:28,937 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:28,937 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:28,939 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:28,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:28,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:28,939 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:28,939 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:28,939 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:28,939 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:28,939 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:28,940 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:28,940 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:28,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:28,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:28,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:28,940 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:28,941 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:28,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:28,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:28,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:28,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:28,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:28,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:28,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:28,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:28,943 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:28,943 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:28,943 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:28,943 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:28,943 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:28,943 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:28,945 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:28,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:28,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:28,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:28,945 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:28,945 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:28,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:28,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:28,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:28,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:28,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:28,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:28,945 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:28,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:28,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:28,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:28,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:28,947 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:28,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:28,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:28,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:28,950 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:28,950 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:28,950 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:28,950 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:28,950 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:28,950 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:28,950 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:28,950 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:28,950 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:28,950 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:28,951 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:28,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:28,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:28,950 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:28,951 DEBUG [Thread-2498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:28,951 DEBUG [Thread-2498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:28,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:28,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:29,051 DEBUG [Thread-2498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:29,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on a5c78075cc0f4fd1864c2f0339ad3d67 2023-03-29 03:55:29,051 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a5c78075cc0f4fd1864c2f0339ad3d67 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-29 03:55:29,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:29,052 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5c78075cc0f4fd1864c2f0339ad3d67, store=A 2023-03-29 03:55:29,052 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:55:29,052 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5c78075cc0f4fd1864c2f0339ad3d67, store=B 2023-03-29 03:55:29,052 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:55:29,052 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5c78075cc0f4fd1864c2f0339ad3d67, store=C 2023-03-29 03:55:29,052 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:55:29,053 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:29,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:29,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:29,055 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:29,055 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:29,055 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:29,055 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:29,056 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:29,056 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:29,057 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:29,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:29,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:29,057 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:29,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:29,058 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:29,059 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:29,059 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:29,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:29,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:29,059 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:29,059 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:29,059 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:29,059 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:29,060 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:29,060 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:29,061 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:29,061 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:29,061 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:29,061 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. started... 2023-03-29 03:55:29,061 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. as already flushing 2023-03-29 03:55:29,061 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:29,061 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:29,062 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:29,062 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:29,062 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:29,062 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:29,064 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:29,064 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:29,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:29,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:29,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:29,065 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:29,065 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:29,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:29,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:29,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:29,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:29,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:29,066 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:29,066 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4122e36f[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:29,066 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:29,066 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:29,068 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:29,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:29,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:29,068 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:29,068 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:29,068 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:29,068 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:29,068 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:29,070 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:29,070 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:29,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:29,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:29,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:29,071 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:29,071 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:29,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:29,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:29,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:29,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:29,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:29,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:29,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:29,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:29,073 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:29,073 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:29,073 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:29,073 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:29,073 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:29,073 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:29,075 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:29,075 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:29,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:29,075 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:29,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:29,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:29,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:29,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:29,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:29,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:29,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:29,076 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:29,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:29,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:29,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:29,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:29,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:29,077 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:29,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:29,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:29,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:29,079 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:29,080 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:29,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:29,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:29,080 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:29,080 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:29,080 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:29,080 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:29,080 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:29,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:29,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:29,080 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:29,080 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:29,080 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:29,080 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:29,080 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:29,080 DEBUG [Thread-2498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:29,080 DEBUG [Thread-2498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:29,119 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680062129090, max timestamp=1680062129117Num uniques 859; 2023-03-29 03:55:29,119 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-29 03:55:29,119 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680062129104, max timestamp=1680062129117Num uniques 859; 2023-03-29 03:55:29,119 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680062129090, max timestamp=1680062129117Num uniques 859; 2023-03-29 03:55:29,119 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-29 03:55:29,119 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-29 03:55:29,179 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:29,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51970 deadline: 1680062189172, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:29,181 DEBUG [Thread-2498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:29,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:29,183 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:29,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:29,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:29,184 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:29,184 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:29,184 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:29,184 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:29,185 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:29,185 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:29,186 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:29,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:29,186 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:29,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:29,187 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:29,187 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:29,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:29,187 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:29,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:29,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:29,189 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:29,189 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:29,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51990 deadline: 1680062189180, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:29,190 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:29,190 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:29,191 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:29,191 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:29,191 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:29,191 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:29,191 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:29,191 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:29,191 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. started... 2023-03-29 03:55:29,191 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. as already flushing 2023-03-29 03:55:29,191 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:29,191 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:29,191 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:29,191 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:29,191 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:29,191 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:29,193 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:29,193 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:29,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:29,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:29,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:29,193 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:29,193 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:29,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:29,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:29,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:29,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:29,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:29,195 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:29,195 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@40eca935[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:29,195 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:29,195 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:29,196 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:29,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:29,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:29,196 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:29,196 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:29,196 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:29,197 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:29,197 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:29,199 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:29,199 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:29,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:29,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:29,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:29,199 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:29,199 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:29,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:29,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:29,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:29,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:29,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:29,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:29,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:29,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:29,201 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:29,202 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:29,202 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:29,202 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:29,202 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:29,202 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:29,205 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:29,205 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:29,205 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:29,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:29,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:29,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:29,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:29,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:29,205 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:29,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:29,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:29,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:29,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:29,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:29,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:29,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:29,207 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:29,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:29,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:29,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:29,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:29,209 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:29,209 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:29,209 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:29,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:29,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:29,209 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:29,210 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:29,209 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:29,210 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:29,210 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:29,210 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:29,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:29,210 DEBUG [Thread-2498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:29,210 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:29,210 DEBUG [Thread-2498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:29,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:29,210 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:29,210 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:29,211 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:55:29,286 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:29,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51970 deadline: 1680062189282, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:29,295 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:29,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51990 deadline: 1680062189291, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:29,310 DEBUG [Thread-2498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:29,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:29,312 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:29,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:29,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:29,313 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:29,313 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:29,313 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:29,313 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:29,314 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:29,314 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:29,315 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:29,315 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:29,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:29,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:29,316 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:29,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:29,316 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:29,316 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:29,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:29,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:29,317 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:29,317 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:29,317 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:29,317 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:29,317 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:29,317 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:29,317 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:29,317 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:29,317 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:29,317 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. started... 2023-03-29 03:55:29,318 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. as already flushing 2023-03-29 03:55:29,318 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:29,318 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:29,318 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:29,318 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:29,318 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:29,318 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:29,327 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:29,327 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:29,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:29,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:29,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:29,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:29,353 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:29,354 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:29,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:29,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:29,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:29,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:29,355 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:29,355 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f6bc182[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:29,356 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:29,356 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:29,357 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:29,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:29,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:29,357 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:29,357 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:29,357 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:29,357 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:29,357 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:29,358 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:29,358 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:29,358 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:29,359 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:29,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:29,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:29,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:29,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:29,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:29,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:29,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:29,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:29,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:29,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:29,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:29,361 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:29,361 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:29,361 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:29,361 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:29,361 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:29,361 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:29,363 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:29,363 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:29,363 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:29,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:29,363 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:29,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:29,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:29,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:29,363 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:29,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:29,363 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:29,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:29,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:29,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:29,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:29,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:29,364 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:29,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:29,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:29,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:29,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:29,368 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:29,368 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:29,368 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:29,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:29,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:29,368 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:29,368 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:55:29,368 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:29,368 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:29,368 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:29,368 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:29,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:29,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:29,368 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:29,368 DEBUG [Thread-2498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:29,368 DEBUG [Thread-2498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:29,368 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:29,368 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:29,368 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:29,465 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230329fef9462da461456bba94701a93985194 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230329fef9462da461456bba94701a93985194 2023-03-29 03:55:29,468 DEBUG [Thread-2498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:29,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:29,471 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:29,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:29,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:29,477 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:29,478 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:29,478 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:29,478 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:29,478 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:29,478 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:29,479 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:29,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:29,479 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:29,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:29,480 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:29,480 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:29,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:29,480 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:29,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:29,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:29,481 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:29,481 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:29,481 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:29,481 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:29,481 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:29,481 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:29,481 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:29,481 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:29,481 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:29,481 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. started... 2023-03-29 03:55:29,481 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. as already flushing 2023-03-29 03:55:29,481 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:29,481 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:29,481 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:29,481 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:29,481 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:29,481 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:29,483 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:29,483 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:29,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:29,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:29,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:29,483 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:29,483 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:29,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:29,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:29,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:29,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:29,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:29,484 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:29,484 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7526c3d2[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:29,484 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:29,484 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:29,487 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:29,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:29,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:29,487 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:29,487 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:29,487 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:29,487 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:29,487 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:29,489 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:29,489 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:29,489 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:29,489 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:29,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:29,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:29,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:29,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:29,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:29,490 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:29,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:29,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51970 deadline: 1680062189488, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:29,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:29,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:29,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:29,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:29,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:29,491 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:29,491 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:29,491 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:29,491 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:29,491 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:29,491 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:29,493 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:29,493 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:29,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:29,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:29,493 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:29,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:29,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:29,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:29,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:29,493 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:29,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:29,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:29,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:29,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:29,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:29,494 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:29,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:29,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:29,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:29,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:29,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:29,496 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:29,496 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:29,496 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:29,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:29,497 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:29,497 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:29,496 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:29,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:29,497 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:29,497 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:55:29,497 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:29,497 DEBUG [Thread-2498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:29,497 DEBUG [Thread-2498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:29,497 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:29,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:29,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:29,497 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:29,497 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:29,497 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:29,503 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:29,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51990 deadline: 1680062189498, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:29,575 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:29,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51946 deadline: 1680062189570, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:29,577 DEBUG [Thread-2494] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8161 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67., hostname=jenkins-hbase4.apache.org,35599,1680062018534, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-29 03:55:29,597 DEBUG [Thread-2498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:29,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:29,599 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:29,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:29,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:29,600 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:29,600 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:29,600 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:29,600 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:29,601 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:29,601 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:29,603 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:29,603 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:29,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:29,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:29,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:29,604 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:29,604 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:29,604 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:29,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:29,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:29,605 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:29,605 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:29,605 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:29,605 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:29,605 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:29,605 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:29,605 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:29,605 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:29,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51994 deadline: 1680062189601, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:29,606 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:29,605 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:29,606 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. started... 2023-03-29 03:55:29,606 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. as already flushing 2023-03-29 03:55:29,606 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:29,607 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:29,607 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:29,607 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:29,607 DEBUG [Thread-2496] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8192 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67., hostname=jenkins-hbase4.apache.org,35599,1680062018534, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-29 03:55:29,607 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:29,607 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:29,609 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:29,609 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:29,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:29,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:29,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:29,609 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:29,609 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:29,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:29,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:29,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:29,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:29,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:29,610 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:29,610 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@23c96f82[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:29,610 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:29,611 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:29,612 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:29,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:29,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:29,612 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:29,612 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:29,612 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:29,613 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:29,613 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:29,614 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:29,614 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:29,614 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:29,614 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:29,614 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:29,614 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:29,614 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:29,614 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:29,614 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:29,614 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:29,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51938 deadline: 1680062189612, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:29,616 DEBUG [Thread-2488] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8202 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67., hostname=jenkins-hbase4.apache.org,35599,1680062018534, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-29 03:55:29,618 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:29,618 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:29,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:29,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:29,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:29,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:29,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:29,618 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:29,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:29,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:29,619 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:29,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:29,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:29,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:29,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:29,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:29,620 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:29,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:29,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:29,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:29,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:29,622 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:29,622 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:29,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:29,622 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:29,622 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:29,622 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:29,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:29,622 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:29,622 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:29,623 DEBUG [Thread-2498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:29,623 DEBUG [Thread-2498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:29,623 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:55:29,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:29,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:29,623 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:29,623 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:29,623 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:29,623 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:29,623 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:29,723 DEBUG [Thread-2498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:29,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:29,725 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:29,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:29,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:29,726 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:29,726 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:29,726 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:29,726 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:29,726 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:29,727 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:29,728 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:29,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:29,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:29,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:29,729 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:29,729 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:29,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:29,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:29,729 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:29,729 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:29,729 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:29,729 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:29,729 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:29,729 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:29,729 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:29,730 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:29,730 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:29,730 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:29,730 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:29,730 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. started... 2023-03-29 03:55:29,730 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. as already flushing 2023-03-29 03:55:29,730 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:29,730 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:29,730 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:29,730 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:29,730 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:29,730 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:29,731 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:29,731 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:29,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:29,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:29,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:29,732 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:29,732 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:29,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:29,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:29,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:29,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:29,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:29,734 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:29,734 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c9dfcf[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:29,734 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:29,734 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:29,736 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:29,736 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:29,737 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:29,737 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:29,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:29,737 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:29,738 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:29,738 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:29,741 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:29,741 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:29,741 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:29,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:29,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:29,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:29,741 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:29,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:29,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:29,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:29,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:29,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:29,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:29,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:29,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:29,744 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:29,744 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:29,744 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:29,744 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:29,744 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:29,744 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:29,746 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:29,746 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:29,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:29,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:29,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:29,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:29,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:29,746 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:29,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:29,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:29,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:29,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:29,747 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:29,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:29,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:29,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:29,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:29,750 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:29,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:29,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:29,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:29,757 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:29,757 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:29,757 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:29,757 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:29,757 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:29,757 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:29,757 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:29,757 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:29,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:29,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:29,757 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:29,757 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:29,757 DEBUG [Thread-2498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:29,757 DEBUG [Thread-2498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:29,757 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:29,757 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:29,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:29,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:29,795 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:29,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51970 deadline: 1680062189791, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:29,814 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:29,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51990 deadline: 1680062189805, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:29,857 DEBUG [Thread-2498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:29,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:29,859 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:29,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:29,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:29,860 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:29,860 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:29,860 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:29,860 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:29,861 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:29,861 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:29,862 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:29,862 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:29,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:29,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:29,862 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:29,862 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:29,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:29,863 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:29,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:29,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:29,863 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:29,863 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:29,863 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:29,864 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:29,864 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:29,864 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:29,864 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:29,864 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:29,864 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:29,864 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. started... 2023-03-29 03:55:29,865 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. as already flushing 2023-03-29 03:55:29,865 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:29,865 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:29,865 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:29,865 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:29,865 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:29,865 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:29,866 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:29,866 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:29,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:29,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:29,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:29,866 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:29,866 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:29,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:29,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:29,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:29,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:29,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:29,868 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:29,868 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:29,868 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@278f6945[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:29,868 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:29,869 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:29,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:29,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:29,869 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:29,869 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:29,869 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:29,869 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:29,869 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:29,871 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:29,871 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:29,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:29,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:29,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:29,871 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:29,872 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:29,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:29,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:29,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:29,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:29,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:29,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:29,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:29,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:29,873 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:29,873 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:29,873 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:29,873 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:29,873 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:29,873 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:29,875 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:29,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:29,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:29,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:29,875 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:29,875 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:29,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:29,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:29,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:29,875 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:29,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:29,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:29,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:29,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:29,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:29,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:29,877 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:29,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:29,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:29,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:29,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:29,879 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:29,879 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:29,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:29,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:29,879 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:29,879 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:29,879 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:29,879 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:29,879 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:29,879 DEBUG [Thread-2498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:29,879 DEBUG [Thread-2498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:29,879 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:29,879 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:29,880 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:29,880 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:29,879 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:29,880 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:55:29,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:29,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:29,881 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=218, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/A/ad7ef9e3d87d40f7accb56d92b20437d 2023-03-29 03:55:29,899 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=218 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/B/be95af63faee4fdbb7ba85abfa3ba828 2023-03-29 03:55:29,979 DEBUG [Thread-2498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:29,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:29,982 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:29,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:29,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:29,983 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:29,983 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:29,983 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:29,983 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:29,984 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:29,984 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:29,985 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:29,985 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:29,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:29,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:29,985 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:29,985 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:29,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:29,986 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:29,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:29,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:29,986 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:29,986 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:29,986 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:29,987 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:29,987 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:29,987 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:29,987 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:29,987 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:29,987 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:29,987 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. started... 2023-03-29 03:55:29,987 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. as already flushing 2023-03-29 03:55:29,987 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:29,987 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:29,987 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:29,987 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:29,987 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:29,987 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:29,988 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:29,988 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:29,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:29,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:29,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:29,989 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:29,989 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:29,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:29,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:29,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:29,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:29,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:29,990 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:29,991 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@64f40635[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:29,991 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:29,991 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:29,992 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:29,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:29,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:29,992 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:29,992 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:29,992 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:29,992 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:29,992 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:29,994 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:29,995 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:29,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:29,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:29,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:29,995 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:29,995 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:29,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:29,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:29,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:29,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:29,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:29,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:29,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:29,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:29,998 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:29,998 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:29,998 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:29,998 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:29,998 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:29,998 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:29,999 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:29,999 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:29,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:30,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:30,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:29,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:30,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:29,999 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:30,000 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:30,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:30,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:30,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:30,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:30,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:30,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:30,001 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:30,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:30,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:30,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:30,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:30,003 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:30,003 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:30,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:30,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:30,003 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:30,003 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:30,003 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:30,003 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:30,003 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:30,003 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:30,003 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:30,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:30,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:30,004 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:55:30,004 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:30,004 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:30,004 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:30,004 DEBUG [Thread-2498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:30,004 DEBUG [Thread-2498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:30,104 DEBUG [Thread-2498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:30,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:30,106 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:30,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:30,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:30,108 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:30,108 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:30,108 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:30,108 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:30,108 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:30,108 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:30,110 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:30,110 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:30,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:30,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:30,110 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:30,110 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:30,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:30,110 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:30,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:30,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:30,111 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:30,111 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:30,111 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:30,111 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:30,111 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:30,111 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:30,111 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:30,111 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:30,111 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:30,112 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. started... 2023-03-29 03:55:30,112 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. as already flushing 2023-03-29 03:55:30,112 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:30,112 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:30,112 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:30,112 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:30,112 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:30,112 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:30,114 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:30,114 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:30,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:30,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:30,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:30,114 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:30,114 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:30,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:30,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:30,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:30,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:30,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:30,115 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:30,116 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ca9c056[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:30,116 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:30,116 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:30,117 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:30,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:30,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:30,117 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:30,117 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:30,117 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:30,117 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:30,117 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:30,118 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:30,118 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:30,118 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:30,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:30,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:30,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:30,118 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:30,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:30,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:30,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:30,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:30,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:30,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:30,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:30,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:30,121 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:30,121 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:30,121 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:30,121 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:30,121 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:30,121 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:30,126 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:30,126 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:30,126 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:30,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:30,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:30,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:30,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:30,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:30,127 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:30,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:30,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:30,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:30,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:30,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:30,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:30,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:30,128 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:30,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:30,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:30,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:30,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:30,133 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:30,133 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:30,133 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:30,133 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:30,133 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:30,133 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:30,133 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:30,133 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:30,133 DEBUG [Thread-2498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:30,133 DEBUG [Thread-2498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:30,133 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:30,133 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:30,134 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:55:30,134 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:30,134 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:30,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:30,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:30,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:30,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:30,233 DEBUG [Thread-2498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:30,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:30,235 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:30,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:30,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:30,236 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:30,236 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:30,236 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:30,237 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:30,237 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:30,237 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:30,238 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:30,238 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:30,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:30,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:30,238 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:30,238 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:30,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:30,239 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:30,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:30,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:30,239 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:30,239 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:30,239 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:30,239 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:30,240 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:30,240 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:30,240 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:30,240 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:30,240 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:30,240 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. started... 2023-03-29 03:55:30,240 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. as already flushing 2023-03-29 03:55:30,240 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:30,240 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:30,240 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:30,240 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:30,240 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:30,240 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:30,243 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:30,243 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:30,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:30,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:30,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:30,243 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:30,243 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:30,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:30,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:30,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:30,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:30,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:30,244 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:30,244 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@44d641ab[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:30,244 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:30,245 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:30,246 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:30,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:30,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:30,246 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:30,246 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:30,246 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:30,246 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:30,246 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:30,248 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:30,248 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:30,248 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:30,248 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:30,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:30,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:30,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:30,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:30,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:30,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:30,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:30,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:30,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:30,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:30,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:30,251 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:30,251 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:30,251 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:30,251 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:30,251 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:30,251 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:30,252 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:30,252 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:30,252 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:30,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:30,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:30,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:30,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:30,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:30,253 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:30,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:30,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:30,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:30,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:30,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:30,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:30,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:30,254 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:30,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:30,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:30,257 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:30,257 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:30,257 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:30,257 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:30,257 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:30,257 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:30,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:30,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:30,257 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:30,258 DEBUG [Thread-2498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:30,257 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:55:30,257 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:30,258 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:30,258 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:30,258 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:30,258 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:30,258 DEBUG [Thread-2498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:30,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:30,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:30,302 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:30,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51970 deadline: 1680062190298, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:30,309 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=218 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/C/52b982e75b9a42089f729d3115dcfadd 2023-03-29 03:55:30,313 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/A/ad7ef9e3d87d40f7accb56d92b20437d as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/ad7ef9e3d87d40f7accb56d92b20437d 2023-03-29 03:55:30,316 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/ad7ef9e3d87d40f7accb56d92b20437d, entries=200, sequenceid=218, filesize=32.1 K 2023-03-29 03:55:30,317 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/B/be95af63faee4fdbb7ba85abfa3ba828 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/be95af63faee4fdbb7ba85abfa3ba828 2023-03-29 03:55:30,320 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/be95af63faee4fdbb7ba85abfa3ba828, entries=150, sequenceid=218, filesize=11.8 K 2023-03-29 03:55:30,321 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/C/52b982e75b9a42089f729d3115dcfadd as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/52b982e75b9a42089f729d3115dcfadd 2023-03-29 03:55:30,322 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:30,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:51990 deadline: 1680062190316, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:30,323 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/52b982e75b9a42089f729d3115dcfadd, entries=150, sequenceid=218, filesize=11.8 K 2023-03-29 03:55:30,324 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=136.42 KB/139690 for a5c78075cc0f4fd1864c2f0339ad3d67 in 1273ms, sequenceid=218, compaction requested=true 2023-03-29 03:55:30,324 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a5c78075cc0f4fd1864c2f0339ad3d67: 2023-03-29 03:55:30,324 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:55:30,324 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-29 03:55:30,325 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-29 03:55:30,324 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:55:30,325 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:55:30,325 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 84986 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-29 03:55:30,325 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1912): a5c78075cc0f4fd1864c2f0339ad3d67/A is initiating minor compaction (all files) 2023-03-29 03:55:30,326 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36416 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-29 03:55:30,326 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a5c78075cc0f4fd1864c2f0339ad3d67/A in TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:30,326 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1912): a5c78075cc0f4fd1864c2f0339ad3d67/B is initiating minor compaction (all files) 2023-03-29 03:55:30,326 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/496176f4abe44d389b39e070052d006f, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/8848dbbad301430ea69d561a16c84f9f, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/ad7ef9e3d87d40f7accb56d92b20437d] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp, totalSize=83.0 K 2023-03-29 03:55:30,326 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a5c78075cc0f4fd1864c2f0339ad3d67/B in TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:30,326 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/449af9b48e734a02b5e692557b1a1e17, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/a2e31e392f844f6698bae1872e0f9b15, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/be95af63faee4fdbb7ba85abfa3ba828] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp, totalSize=35.6 K 2023-03-29 03:55:30,326 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 496176f4abe44d389b39e070052d006f, keycount=150, bloomtype=ROW, size=25.6 K, encoding=NONE, compression=NONE, seqNum=186, earliestPutTs=1680062126190 2023-03-29 03:55:30,326 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 449af9b48e734a02b5e692557b1a1e17, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=186, earliestPutTs=1680062126190 2023-03-29 03:55:30,326 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 8848dbbad301430ea69d561a16c84f9f, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=201, earliestPutTs=1680062126842 2023-03-29 03:55:30,327 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting a2e31e392f844f6698bae1872e0f9b15, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=201, earliestPutTs=1680062126842 2023-03-29 03:55:30,327 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting ad7ef9e3d87d40f7accb56d92b20437d, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=218, earliestPutTs=1680062129043 2023-03-29 03:55:30,327 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting be95af63faee4fdbb7ba85abfa3ba828, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=218, earliestPutTs=1680062129043 2023-03-29 03:55:30,333 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] throttle.PressureAwareThroughputController(145): a5c78075cc0f4fd1864c2f0339ad3d67#B#compaction#377 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:55:30,335 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] throttle.PressureAwareThroughputController(145): a5c78075cc0f4fd1864c2f0339ad3d67#A#compaction#376 average throughput is 4.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:55:30,352 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/B/4ef04e1fe5da4761afd2ece4d4fc9d1e as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/4ef04e1fe5da4761afd2ece4d4fc9d1e 2023-03-29 03:55:30,355 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a5c78075cc0f4fd1864c2f0339ad3d67/B of a5c78075cc0f4fd1864c2f0339ad3d67 into 4ef04e1fe5da4761afd2ece4d4fc9d1e(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:55:30,355 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a5c78075cc0f4fd1864c2f0339ad3d67: 2023-03-29 03:55:30,356 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67., storeName=a5c78075cc0f4fd1864c2f0339ad3d67/B, priority=13, startTime=1680062130324; duration=0sec 2023-03-29 03:55:30,356 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:55:30,356 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-29 03:55:30,357 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36416 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-29 03:55:30,357 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1912): a5c78075cc0f4fd1864c2f0339ad3d67/C is initiating minor compaction (all files) 2023-03-29 03:55:30,357 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a5c78075cc0f4fd1864c2f0339ad3d67/C in TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:30,357 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/5e7b29bba65940acabd3c8d28c38897d, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/d0ed71e2627849f195cefc1eefb1ff72, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/52b982e75b9a42089f729d3115dcfadd] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp, totalSize=35.6 K 2023-03-29 03:55:30,357 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 5e7b29bba65940acabd3c8d28c38897d, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=186, earliestPutTs=1680062126190 2023-03-29 03:55:30,357 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting d0ed71e2627849f195cefc1eefb1ff72, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=201, earliestPutTs=1680062126842 2023-03-29 03:55:30,357 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 52b982e75b9a42089f729d3115dcfadd, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=218, earliestPutTs=1680062129043 2023-03-29 03:55:30,358 DEBUG [Thread-2498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:30,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:30,360 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:30,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:30,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:30,362 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:30,362 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:30,362 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:30,362 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:30,363 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:30,363 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:30,364 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:30,364 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:30,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:30,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:30,365 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:30,365 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:30,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:30,365 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:30,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:30,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:30,365 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:30,366 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:30,366 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:30,366 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] throttle.PressureAwareThroughputController(145): a5c78075cc0f4fd1864c2f0339ad3d67#C#compaction#378 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:55:30,366 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:30,366 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:30,366 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:30,388 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:30,388 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:30,388 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:30,388 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. started... 2023-03-29 03:55:30,388 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a5c78075cc0f4fd1864c2f0339ad3d67 3/3 column families, dataSize=136.42 KB heapSize=385.38 KB 2023-03-29 03:55:30,390 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5c78075cc0f4fd1864c2f0339ad3d67, store=A 2023-03-29 03:55:30,390 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-29 03:55:30,390 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5c78075cc0f4fd1864c2f0339ad3d67, store=B 2023-03-29 03:55:30,390 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-29 03:55:30,391 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5c78075cc0f4fd1864c2f0339ad3d67, store=C 2023-03-29 03:55:30,391 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-29 03:55:30,429 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/C/e89c41ae679d43748bade17b8b1a1144 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/e89c41ae679d43748bade17b8b1a1144 2023-03-29 03:55:30,436 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a5c78075cc0f4fd1864c2f0339ad3d67/C of a5c78075cc0f4fd1864c2f0339ad3d67 into e89c41ae679d43748bade17b8b1a1144(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:55:30,436 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a5c78075cc0f4fd1864c2f0339ad3d67: 2023-03-29 03:55:30,436 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67., storeName=a5c78075cc0f4fd1864c2f0339ad3d67/C, priority=13, startTime=1680062130325; duration=0sec 2023-03-29 03:55:30,437 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:55:30,442 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303299e5576e0b93c422bb557a58db2a90df6 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303299e5576e0b93c422bb557a58db2a90df6 2023-03-29 03:55:30,460 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=255, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/A/cf6aa67f579649e7b0fe9f2277172a14 2023-03-29 03:55:30,482 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=255 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/B/5860d676a0af4102a1ab89c009bafdfb 2023-03-29 03:55:30,946 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=255 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/C/15dee25bd8cd419c9c0971ac2d17662f 2023-03-29 03:55:30,949 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/A/cf6aa67f579649e7b0fe9f2277172a14 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/cf6aa67f579649e7b0fe9f2277172a14 2023-03-29 03:55:30,953 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/cf6aa67f579649e7b0fe9f2277172a14, entries=150, sequenceid=255, filesize=25.3 K 2023-03-29 03:55:30,953 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/B/5860d676a0af4102a1ab89c009bafdfb as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/5860d676a0af4102a1ab89c009bafdfb 2023-03-29 03:55:30,956 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/5860d676a0af4102a1ab89c009bafdfb, entries=150, sequenceid=255, filesize=11.8 K 2023-03-29 03:55:30,957 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/C/15dee25bd8cd419c9c0971ac2d17662f as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/15dee25bd8cd419c9c0971ac2d17662f 2023-03-29 03:55:30,960 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/15dee25bd8cd419c9c0971ac2d17662f, entries=150, sequenceid=255, filesize=11.8 K 2023-03-29 03:55:30,961 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~136.42 KB/139690, heapSize ~385.33 KB/394576, currentSize=0 B/0 for a5c78075cc0f4fd1864c2f0339ad3d67 in 573ms, sequenceid=255, compaction requested=false 2023-03-29 03:55:30,961 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a5c78075cc0f4fd1864c2f0339ad3d67: 2023-03-29 03:55:30,961 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:30,961 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:30,961 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:30,961 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:30,962 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:30,962 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:30,963 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:30,963 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:30,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:30,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:30,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:30,963 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:30,963 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:30,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:30,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:30,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:30,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:30,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:30,965 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:30,965 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@184fdd35[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:30,965 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:30,965 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:30,966 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:30,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:30,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:30,967 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:30,967 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:30,967 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:30,967 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:30,967 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:30,968 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:30,968 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:30,968 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:30,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:30,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:30,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:30,968 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:30,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:30,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:30,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:30,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:30,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:30,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:30,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:30,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:30,971 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:30,971 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:30,971 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:30,971 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:30,971 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:30,971 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:30,974 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:30,974 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:30,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:30,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:30,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:30,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:30,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:30,974 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:30,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:30,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:30,974 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:30,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:30,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:30,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:30,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:30,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:30,976 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:30,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:30,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:30,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:30,978 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:30,978 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:30,978 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:30,978 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:30,978 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:30,978 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:30,978 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:30,978 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:30,978 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:30,978 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:30,978 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:30,978 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:55:30,978 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:30,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:30,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:30,978 DEBUG [Thread-2498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:30,978 DEBUG [Thread-2498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:30,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:30,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:31,078 DEBUG [Thread-2498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:31,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:31,081 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:31,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:31,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:31,082 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:31,082 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:31,082 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:31,082 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:31,082 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:31,082 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:31,084 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:31,084 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:31,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:31,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:31,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:31,085 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:31,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:31,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:31,086 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:31,086 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:31,086 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:31,088 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:31,088 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:31,088 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:31,088 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:31,088 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:31,088 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:31,088 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:31,088 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:31,088 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. started... 2023-03-29 03:55:31,088 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a5c78075cc0f4fd1864c2f0339ad3d67: 2023-03-29 03:55:31,088 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:31,088 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:31,089 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:31,089 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:31,089 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:31,089 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:31,090 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:31,090 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:31,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:31,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:31,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:31,090 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:31,090 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:31,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:31,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:31,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:31,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:31,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:31,092 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:31,092 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:31,092 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:31,092 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c4ea5a7[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:31,093 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:31,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:31,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:31,094 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:31,094 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:31,094 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:31,094 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:31,094 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:31,095 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:31,095 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:31,095 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:31,095 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:31,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:31,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:31,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:31,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:31,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:31,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:31,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:31,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:31,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:31,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:31,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:31,098 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:31,098 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:31,098 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:31,098 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:31,098 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:31,098 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:31,100 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:31,100 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:31,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:31,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:31,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:31,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:31,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:31,100 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:31,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:31,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:31,101 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:31,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:31,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:31,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:31,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:31,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:31,102 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:31,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:31,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:31,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:31,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:31,104 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:31,104 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:31,104 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:31,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:31,105 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:31,105 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:31,105 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:31,105 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:31,104 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:31,105 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:31,105 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:55:31,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:31,105 DEBUG [Thread-2498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:31,105 DEBUG [Thread-2498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:31,105 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:31,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:31,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:31,105 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:31,105 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:31,205 DEBUG [Thread-2498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:31,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:31,207 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:31,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:31,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:31,208 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:31,209 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:31,209 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:31,209 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:31,209 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:31,209 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:31,211 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:31,211 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:31,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:31,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:31,213 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:31,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:31,213 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:31,213 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:31,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:31,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:31,213 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:31,214 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:31,214 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:31,214 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:31,214 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:31,214 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:31,214 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:31,214 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:31,214 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:31,214 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. started... 2023-03-29 03:55:31,214 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a5c78075cc0f4fd1864c2f0339ad3d67: 2023-03-29 03:55:31,214 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:31,214 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:31,214 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:31,214 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:31,214 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:31,215 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:31,216 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:31,216 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:31,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:31,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:31,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:31,216 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:31,216 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:31,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:31,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:31,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:31,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:31,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:31,218 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:31,218 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e277508[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:31,218 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:31,218 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:31,219 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:31,219 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:31,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:31,219 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:31,219 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:31,219 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:31,219 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:31,219 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:31,221 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:31,221 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:31,221 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:31,221 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:31,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:31,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:31,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:31,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:31,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:31,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:31,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:31,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:31,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:31,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:31,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:31,224 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:31,224 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:31,224 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:31,224 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:31,224 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:31,224 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:31,227 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:31,227 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:31,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:31,227 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:31,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:31,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:31,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:31,227 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:31,227 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:31,227 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:31,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:31,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:31,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:31,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:31,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:31,228 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:31,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:31,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:31,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:31,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:31,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:31,231 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:31,231 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:31,231 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:31,231 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:31,231 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:31,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:31,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:31,231 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:31,231 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:31,231 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:31,231 DEBUG [Thread-2498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:31,231 DEBUG [Thread-2498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:31,231 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:55:31,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:31,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:31,231 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:31,232 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:31,232 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:31,232 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:31,331 DEBUG [Thread-2498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:31,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:31,334 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:31,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:31,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:31,335 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:31,335 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:31,335 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:31,335 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:31,335 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:31,335 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:31,336 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:31,336 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:31,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:31,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:31,337 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:31,337 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:31,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:31,337 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:31,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:31,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:31,338 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:31,338 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:31,338 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:31,338 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:31,338 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:31,338 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:31,338 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:31,338 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:31,338 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:31,338 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. started... 2023-03-29 03:55:31,339 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a5c78075cc0f4fd1864c2f0339ad3d67: 2023-03-29 03:55:31,339 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:31,339 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:31,339 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:31,339 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:31,339 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:31,339 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:31,340 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:31,340 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:31,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:31,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:31,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:31,340 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:31,340 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:31,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:31,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:31,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:31,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:31,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:31,342 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:31,342 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@688aa47e[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:31,342 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:31,342 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:31,344 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:31,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:31,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:31,344 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:31,344 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:31,345 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:31,344 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:31,345 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:31,346 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:31,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:31,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:31,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:31,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:31,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:31,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:31,347 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:31,347 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:31,347 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:31,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:31,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:31,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:31,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:31,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:31,349 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:31,349 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:31,349 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:31,349 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:31,349 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:31,349 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:31,351 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:31,351 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:31,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:31,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:31,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:31,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:31,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:31,351 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:31,351 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:31,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:31,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:31,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:31,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:31,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:31,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:31,352 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:31,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:31,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:31,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:31,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:31,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:31,354 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:31,354 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:31,354 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:31,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:31,354 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:31,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:31,354 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:31,355 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:31,354 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:31,355 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:31,355 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:31,355 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:31,355 DEBUG [Thread-2498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:31,354 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:31,355 DEBUG [Thread-2498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:31,355 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:31,355 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:55:31,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:31,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:31,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on a5c78075cc0f4fd1864c2f0339ad3d67 2023-03-29 03:55:31,414 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a5c78075cc0f4fd1864c2f0339ad3d67 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-29 03:55:31,415 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5c78075cc0f4fd1864c2f0339ad3d67, store=A 2023-03-29 03:55:31,415 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:55:31,415 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5c78075cc0f4fd1864c2f0339ad3d67, store=B 2023-03-29 03:55:31,415 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:55:31,415 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5c78075cc0f4fd1864c2f0339ad3d67, store=C 2023-03-29 03:55:31,415 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:55:31,426 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303296460d796f8e945c78aa3e9c71a325a68 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303296460d796f8e945c78aa3e9c71a325a68 2023-03-29 03:55:31,430 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=269, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/A/5f65fa1579ec4cc6894a148c8d3fa060 2023-03-29 03:55:31,445 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=269 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/B/e266a7c7073742c3a0f270bb8ed0bc41 2023-03-29 03:55:31,455 DEBUG [Thread-2498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:31,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:31,457 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:31,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:31,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:31,466 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:31,466 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:31,466 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:31,466 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:31,468 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:31,468 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:31,468 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=269 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/C/fdd78514d3564102ba146d3c7df2cc02 2023-03-29 03:55:31,470 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:31,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:31,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:31,470 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:31,470 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:31,471 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:31,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:31,471 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:31,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:31,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:31,471 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:31,472 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/A/5f65fa1579ec4cc6894a148c8d3fa060 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/5f65fa1579ec4cc6894a148c8d3fa060 2023-03-29 03:55:31,476 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/5f65fa1579ec4cc6894a148c8d3fa060, entries=300, sequenceid=269, filesize=46.1 K 2023-03-29 03:55:31,476 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/B/e266a7c7073742c3a0f270bb8ed0bc41 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/e266a7c7073742c3a0f270bb8ed0bc41 2023-03-29 03:55:31,480 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/e266a7c7073742c3a0f270bb8ed0bc41, entries=150, sequenceid=269, filesize=11.9 K 2023-03-29 03:55:31,480 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:31,480 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:31,480 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:31,480 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:31,480 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:31,480 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:31,480 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:31,480 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:31,480 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/C/fdd78514d3564102ba146d3c7df2cc02 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/fdd78514d3564102ba146d3c7df2cc02 2023-03-29 03:55:31,480 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. started... 2023-03-29 03:55:31,481 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. as already flushing 2023-03-29 03:55:31,481 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:31,481 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:31,481 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:31,481 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:31,481 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:31,481 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:31,483 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:31,483 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:31,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:31,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:31,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:31,483 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:31,483 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:31,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:31,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:31,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:31,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:31,484 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/fdd78514d3564102ba146d3c7df2cc02, entries=150, sequenceid=269, filesize=11.9 K 2023-03-29 03:55:31,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:31,485 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=107.34 KB/109920 for a5c78075cc0f4fd1864c2f0339ad3d67 in 71ms, sequenceid=269, compaction requested=true 2023-03-29 03:55:31,485 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:31,485 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a5c78075cc0f4fd1864c2f0339ad3d67: 2023-03-29 03:55:31,485 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:31,485 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@eca6a2f[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:31,485 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:55:31,485 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:31,485 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:55:31,485 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 3 compacting, 2 eligible, 16 blocking 2023-03-29 03:55:31,485 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-29 03:55:31,486 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-03-29 03:55:31,486 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-03-29 03:55:31,486 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. because compaction request was cancelled 2023-03-29 03:55:31,486 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-29 03:55:31,487 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:31,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:31,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:31,487 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:31,487 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:31,487 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:31,487 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:31,487 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:31,488 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36668 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-29 03:55:31,488 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1912): a5c78075cc0f4fd1864c2f0339ad3d67/B is initiating minor compaction (all files) 2023-03-29 03:55:31,488 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a5c78075cc0f4fd1864c2f0339ad3d67/B in TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:31,488 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/4ef04e1fe5da4761afd2ece4d4fc9d1e, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/5860d676a0af4102a1ab89c009bafdfb, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/e266a7c7073742c3a0f270bb8ed0bc41] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp, totalSize=35.8 K 2023-03-29 03:55:31,488 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 4ef04e1fe5da4761afd2ece4d4fc9d1e, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=218, earliestPutTs=1680062129043 2023-03-29 03:55:31,488 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:31,488 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:31,488 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:31,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:31,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:31,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:31,488 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:31,488 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 5860d676a0af4102a1ab89c009bafdfb, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=255, earliestPutTs=1680062129145 2023-03-29 03:55:31,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:31,489 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting e266a7c7073742c3a0f270bb8ed0bc41, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=269, earliestPutTs=1680062131398 2023-03-29 03:55:31,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:31,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:31,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:31,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:31,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:31,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on a5c78075cc0f4fd1864c2f0339ad3d67 2023-03-29 03:55:31,492 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a5c78075cc0f4fd1864c2f0339ad3d67 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-29 03:55:31,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:31,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:31,493 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:31,493 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5c78075cc0f4fd1864c2f0339ad3d67, store=A 2023-03-29 03:55:31,493 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:31,493 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:31,493 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:31,493 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-29 03:55:31,494 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5c78075cc0f4fd1864c2f0339ad3d67, store=B 2023-03-29 03:55:31,494 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-29 03:55:31,494 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5c78075cc0f4fd1864c2f0339ad3d67, store=C 2023-03-29 03:55:31,494 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-29 03:55:31,493 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:31,494 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:31,498 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] throttle.PressureAwareThroughputController(145): a5c78075cc0f4fd1864c2f0339ad3d67#B#compaction#385 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:55:31,500 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:31,500 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:31,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:31,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:31,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:31,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:31,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:31,504 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:31,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:31,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:31,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:31,505 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:31,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:31,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:31,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:31,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:31,506 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:31,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:31,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:31,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:31,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:31,509 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:31,509 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:31,509 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:31,509 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:31,509 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:31,509 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:31,509 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:31,509 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:31,509 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:31,509 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:31,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:31,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:31,509 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:31,509 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:31,509 DEBUG [Thread-2498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:31,510 DEBUG [Thread-2498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:31,510 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:55:31,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:31,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:31,552 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:31,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51990 deadline: 1680062191547, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:31,553 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:31,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51970 deadline: 1680062191547, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:31,566 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/A/d736a34884574dada571d2b9d7c1b07c as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/d736a34884574dada571d2b9d7c1b07c 2023-03-29 03:55:31,570 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a5c78075cc0f4fd1864c2f0339ad3d67/A of a5c78075cc0f4fd1864c2f0339ad3d67 into d736a34884574dada571d2b9d7c1b07c(size=25.7 K), total size for store is 97.0 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-03-29 03:55:31,570 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a5c78075cc0f4fd1864c2f0339ad3d67: 2023-03-29 03:55:31,570 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67., storeName=a5c78075cc0f4fd1864c2f0339ad3d67/A, priority=13, startTime=1680062130324; duration=1sec 2023-03-29 03:55:31,570 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:55:31,570 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-29 03:55:31,571 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36668 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-29 03:55:31,571 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1912): a5c78075cc0f4fd1864c2f0339ad3d67/C is initiating minor compaction (all files) 2023-03-29 03:55:31,571 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a5c78075cc0f4fd1864c2f0339ad3d67/C in TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:31,571 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/e89c41ae679d43748bade17b8b1a1144, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/15dee25bd8cd419c9c0971ac2d17662f, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/fdd78514d3564102ba146d3c7df2cc02] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp, totalSize=35.8 K 2023-03-29 03:55:31,571 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting e89c41ae679d43748bade17b8b1a1144, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=218, earliestPutTs=1680062129043 2023-03-29 03:55:31,571 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 15dee25bd8cd419c9c0971ac2d17662f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=255, earliestPutTs=1680062129145 2023-03-29 03:55:31,572 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting fdd78514d3564102ba146d3c7df2cc02, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=269, earliestPutTs=1680062131398 2023-03-29 03:55:31,578 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] throttle.PressureAwareThroughputController(145): a5c78075cc0f4fd1864c2f0339ad3d67#C#compaction#387 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:55:31,610 DEBUG [Thread-2498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:31,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:31,612 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:31,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:31,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:31,613 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:31,613 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:31,613 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:31,613 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:31,613 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:31,614 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:31,617 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:31,617 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:31,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:31,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:31,619 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:31,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:31,619 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:31,619 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:31,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:31,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:31,619 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:31,620 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:31,620 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:31,620 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:31,620 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:31,620 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:31,620 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:31,620 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. started... 2023-03-29 03:55:31,620 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:31,620 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. as already flushing 2023-03-29 03:55:31,620 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:31,620 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:31,620 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:31,620 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:31,620 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:31,620 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:31,622 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:31,622 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:31,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:31,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:31,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:31,623 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:31,623 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:31,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:31,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:31,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:31,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:31,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:31,624 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:31,624 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@278095cc[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:31,624 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:31,624 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:31,625 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:31,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:31,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:31,625 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:31,625 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:31,625 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:31,625 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:31,625 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:31,627 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:31,627 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:31,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:31,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:31,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:31,627 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:31,627 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:31,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:31,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:31,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:31,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:31,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:31,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:31,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:31,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:31,629 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:31,629 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:31,630 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:31,630 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:31,630 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:31,630 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:31,631 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:31,631 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:31,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:31,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:31,631 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:31,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:31,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:31,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:31,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:31,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:31,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:31,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:31,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:31,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:31,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:31,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:31,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:31,633 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:31,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:31,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:31,634 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:31,637 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:31,637 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:31,637 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:31,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:31,637 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:31,637 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:31,637 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:31,637 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:31,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:31,637 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:31,638 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:31,638 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:31,638 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:31,638 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:31,638 DEBUG [Thread-2498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:31,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:31,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:31,638 DEBUG [Thread-2498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:31,658 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:31,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51990 deadline: 1680062191653, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:31,658 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:31,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51970 deadline: 1680062191654, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:31,738 DEBUG [Thread-2498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:31,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:31,740 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:31,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:31,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:31,741 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:31,741 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:31,741 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:31,741 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:31,742 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:31,742 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:31,743 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:31,743 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:31,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:31,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:31,743 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:31,743 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:31,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:31,744 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:31,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:31,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:31,744 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:31,744 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:31,744 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:31,744 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:31,744 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:31,744 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:31,744 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:31,744 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:31,745 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:31,745 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. started... 2023-03-29 03:55:31,745 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. as already flushing 2023-03-29 03:55:31,745 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:31,745 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:31,745 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:31,745 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:31,745 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:31,745 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:31,746 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:31,746 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:31,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:31,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:31,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:31,746 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:31,746 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:31,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:31,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:31,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:31,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:31,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:31,747 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:31,747 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:31,747 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2aa2b01[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:31,748 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:31,749 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:31,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:31,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:31,749 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:31,749 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:31,749 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:31,749 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:31,749 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:31,751 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:31,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:31,751 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:31,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:31,751 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:31,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:31,751 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:31,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:31,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:31,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:31,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:31,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:31,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:31,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:31,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:31,753 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:31,753 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:31,753 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:31,753 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:31,753 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:31,753 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:31,755 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:31,755 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:31,755 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:31,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:31,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:31,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:31,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:31,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:31,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:31,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:31,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:31,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:31,757 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:31,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:31,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:31,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:31,758 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:31,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:31,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:31,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:31,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:31,760 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:31,760 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:31,760 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:31,760 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:31,760 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:31,760 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:31,760 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:31,760 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:31,760 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:31,760 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:31,760 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:31,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:31,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:31,760 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:55:31,760 DEBUG [Thread-2498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:31,760 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:31,760 DEBUG [Thread-2498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:31,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:31,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:31,861 DEBUG [Thread-2498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:31,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:31,862 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:31,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:31,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:31,864 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:31,864 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:31,864 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:31,864 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:31,864 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:31,864 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:31,864 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:31,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51970 deadline: 1680062191859, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:31,865 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:31,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51990 deadline: 1680062191859, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:31,866 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:31,866 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:31,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:31,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:31,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:31,866 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:31,866 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:31,867 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:31,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:31,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:31,867 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:31,867 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:31,867 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:31,867 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:31,867 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:31,867 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:31,867 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:31,867 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:31,867 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:31,868 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. started... 2023-03-29 03:55:31,868 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. as already flushing 2023-03-29 03:55:31,868 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:31,868 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:31,868 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:31,868 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:31,868 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:31,868 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:31,871 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:31,871 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:31,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:31,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:31,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:31,871 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:31,871 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:31,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:31,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:31,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:31,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:31,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:31,873 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:31,873 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3fbec67e[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:31,873 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:31,873 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:31,874 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:31,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:31,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:31,874 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:31,874 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:31,874 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:31,874 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:31,874 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:31,875 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:31,875 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:31,875 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:31,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:31,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:31,875 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:31,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:31,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:31,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:31,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:31,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:31,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:31,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:31,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:31,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:31,877 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:31,877 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:31,878 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:31,878 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:31,878 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:31,878 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:31,879 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:31,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:31,879 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:31,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:31,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:31,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:31,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:31,879 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:31,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:31,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:31,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:31,880 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:31,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:31,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:31,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:31,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:31,881 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:31,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:31,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:31,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:31,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:31,883 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:31,883 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:31,883 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:31,884 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:31,884 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:31,884 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:31,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:31,884 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:31,884 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:31,884 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:55:31,884 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:31,884 DEBUG [Thread-2498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:31,884 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:31,884 DEBUG [Thread-2498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:31,884 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:31,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:31,884 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:31,884 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:31,884 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:31,918 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/B/c5866dee61e44f8f914008097aa0e9e1 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/c5866dee61e44f8f914008097aa0e9e1 2023-03-29 03:55:31,919 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230329b68f687e27734966b85d725b421c9fe1 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230329b68f687e27734966b85d725b421c9fe1 2023-03-29 03:55:31,924 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a5c78075cc0f4fd1864c2f0339ad3d67/B of a5c78075cc0f4fd1864c2f0339ad3d67 into c5866dee61e44f8f914008097aa0e9e1(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:55:31,924 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a5c78075cc0f4fd1864c2f0339ad3d67: 2023-03-29 03:55:31,924 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67., storeName=a5c78075cc0f4fd1864c2f0339ad3d67/B, priority=13, startTime=1680062131485; duration=0sec 2023-03-29 03:55:31,924 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:55:31,985 DEBUG [Thread-2498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:31,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:31,987 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:31,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:31,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:31,989 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:31,989 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:31,989 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:31,989 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:31,990 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:31,990 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:31,992 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/C/3285c402839d452589d36b939084301f as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/3285c402839d452589d36b939084301f 2023-03-29 03:55:31,992 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:31,992 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:31,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:31,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:31,993 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:31,993 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:31,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:31,994 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:31,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:31,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:31,994 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:31,996 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:31,996 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:31,996 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:31,996 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:31,996 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:32,000 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a5c78075cc0f4fd1864c2f0339ad3d67/C of a5c78075cc0f4fd1864c2f0339ad3d67 into 3285c402839d452589d36b939084301f(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:55:32,000 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a5c78075cc0f4fd1864c2f0339ad3d67: 2023-03-29 03:55:32,000 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67., storeName=a5c78075cc0f4fd1864c2f0339ad3d67/C, priority=13, startTime=1680062131485; duration=0sec 2023-03-29 03:55:32,000 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:55:32,006 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:32,006 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:32,006 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:32,006 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. started... 2023-03-29 03:55:32,006 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. as already flushing 2023-03-29 03:55:32,006 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:32,006 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:32,006 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:32,006 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:32,006 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:32,006 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:32,011 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:32,011 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:32,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:32,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:32,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:32,011 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:32,011 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:32,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:32,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:32,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:32,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:32,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:32,013 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:32,013 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@627848b4[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:32,013 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:32,013 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:32,015 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:32,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:32,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:32,015 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:32,015 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:32,015 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:32,015 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:32,015 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:32,017 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:32,017 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:32,017 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:32,017 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:32,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:32,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:32,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:32,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:32,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:32,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:32,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:32,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:32,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:32,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:32,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:32,022 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:32,022 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:32,023 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:32,023 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:32,023 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:32,023 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:32,032 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:32,032 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:32,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:32,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:32,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:32,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:32,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:32,033 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:32,032 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:32,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:32,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:32,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:32,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:32,037 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:32,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:32,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:32,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:32,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:32,042 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:32,043 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:32,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:32,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:32,043 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:32,043 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:32,043 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:32,043 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:32,043 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:32,043 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:32,043 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:32,043 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:32,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:32,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:32,043 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:32,043 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:32,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:32,044 DEBUG [Thread-2498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:32,044 DEBUG [Thread-2498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:32,045 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:55:32,144 DEBUG [Thread-2498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:32,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:32,148 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:32,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:32,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:32,152 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:32,152 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:32,152 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:32,152 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:32,153 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:32,153 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:32,154 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:32,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:32,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:32,154 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:32,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:32,155 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:32,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:32,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:32,155 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:32,156 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:32,156 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:32,156 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:32,156 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:32,156 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:32,156 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:32,156 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:32,156 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:32,156 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:32,156 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:32,156 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. started... 2023-03-29 03:55:32,156 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. as already flushing 2023-03-29 03:55:32,156 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:32,156 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:32,156 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:32,156 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:32,156 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:32,156 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:32,158 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:32,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:32,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:32,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:32,158 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:32,159 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:32,159 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:32,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:32,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:32,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:32,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:32,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:32,160 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:32,160 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@41759fc2[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:32,160 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:32,160 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:32,162 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:32,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:32,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:32,162 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:32,162 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:32,162 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:32,163 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:32,163 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:32,165 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:32,166 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:32,166 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:32,166 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:32,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:32,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:32,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:32,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:32,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:32,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:32,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:32,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:32,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:32,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:32,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:32,170 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:32,171 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:32,171 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:32,171 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:32,171 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:32,171 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:32,172 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:32,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51990 deadline: 1680062192167, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:32,172 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:32,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51970 deadline: 1680062192167, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:32,173 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:32,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:32,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:32,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:32,173 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:32,173 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:32,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:32,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:32,173 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:32,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:32,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:32,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:32,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:32,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:32,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:32,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:32,175 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:32,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:32,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:32,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:32,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:32,178 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:32,178 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:32,178 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:32,178 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:32,178 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:32,178 DEBUG [Thread-2498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:32,178 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:32,178 DEBUG [Thread-2498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:32,178 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:32,178 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:32,178 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:32,179 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:32,179 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:32,179 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:32,178 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:55:32,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:32,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:32,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:32,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:32,278 DEBUG [Thread-2498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:32,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:32,282 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:32,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:32,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:32,283 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:32,283 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:32,283 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:32,283 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:32,283 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:32,283 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:32,284 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:32,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:32,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:32,284 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:32,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:32,285 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:32,285 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:32,285 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:32,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:32,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:32,285 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:32,286 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:32,286 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:32,286 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:32,286 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:32,286 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:32,286 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:32,286 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:32,286 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:32,286 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. started... 2023-03-29 03:55:32,286 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. as already flushing 2023-03-29 03:55:32,286 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:32,286 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:32,286 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:32,286 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:32,286 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:32,286 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:32,288 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:32,288 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:32,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:32,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:32,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:32,288 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:32,288 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:32,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:32,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:32,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:32,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:32,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:32,290 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:32,290 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@125446ce[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:32,290 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:32,290 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:32,291 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:32,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:32,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:32,291 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:32,291 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:32,291 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:32,291 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:32,291 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:32,293 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:32,293 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:32,293 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:32,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:32,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:32,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:32,293 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:32,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:32,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:32,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:32,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:32,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:32,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:32,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:32,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:32,295 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:32,295 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:32,295 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:32,295 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:32,295 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:32,296 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:32,297 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:32,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:32,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:32,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:32,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:32,298 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:32,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:32,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:32,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:32,299 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:32,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:32,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:32,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:32,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:32,300 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:32,301 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:32,301 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:32,301 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:32,301 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:32,301 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:32,301 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:32,301 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:32,301 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:32,301 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:32,301 DEBUG [Thread-2498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:32,301 DEBUG [Thread-2498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:32,301 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:55:32,304 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:32,304 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:32,304 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:32,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:32,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:32,305 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:55:32,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:32,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:32,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:32,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:32,325 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=290, memsize=40.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/A/8cf0f8e20dd94b87b97107cb27a9a2e9 2023-03-29 03:55:32,401 DEBUG [Thread-2498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:32,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:32,403 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:32,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:32,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:32,404 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:32,404 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:32,404 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:32,404 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:32,405 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:32,405 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:32,407 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:32,407 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:32,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:32,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:32,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:32,408 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:32,408 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:32,408 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:32,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:32,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:32,409 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:32,409 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:32,409 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:32,409 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:32,409 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:32,409 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:32,409 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:32,409 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:32,409 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:32,409 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. started... 2023-03-29 03:55:32,409 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. as already flushing 2023-03-29 03:55:32,409 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:32,409 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:32,409 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:32,409 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:32,409 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:32,409 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:32,411 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:32,411 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:32,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:32,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:32,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:32,411 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:32,411 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:32,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:32,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:32,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:32,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:32,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:32,413 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:32,413 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:32,413 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b10efcc[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:32,413 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:32,414 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:32,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:32,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:32,414 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:32,414 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:32,414 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:32,414 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:32,414 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:32,419 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:32,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:32,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:32,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:32,419 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:32,419 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:32,419 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:32,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:32,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:32,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:32,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:32,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:32,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:32,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:32,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:32,421 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:32,421 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:32,421 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:32,421 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:32,421 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:32,421 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:32,422 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:32,422 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:32,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:32,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:32,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:32,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:32,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:32,422 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:32,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:32,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:32,423 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:32,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:32,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:32,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:32,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:32,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:32,424 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:32,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:32,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:32,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:32,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:32,427 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:32,427 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:32,427 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:32,427 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:32,427 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:32,427 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:32,427 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:32,427 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:32,427 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:32,427 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:32,427 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:32,428 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:55:32,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:32,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:32,427 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:32,428 DEBUG [Thread-2498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:32,428 DEBUG [Thread-2498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:32,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:32,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:32,528 DEBUG [Thread-2498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:32,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:32,530 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:32,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:32,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:32,531 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:32,532 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:32,532 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:32,532 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:32,532 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:32,532 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:32,534 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:32,534 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:32,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:32,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:32,534 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:32,534 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:32,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:32,534 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:32,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:32,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:32,535 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:32,535 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:32,535 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:32,535 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:32,535 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:32,535 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:32,535 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:32,535 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:32,536 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:32,536 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. started... 2023-03-29 03:55:32,536 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. as already flushing 2023-03-29 03:55:32,536 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:32,536 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:32,536 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:32,536 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:32,536 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:32,536 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:32,543 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:32,543 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:32,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:32,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:32,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:32,544 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:32,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:32,544 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:32,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:32,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:32,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:32,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:32,545 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:32,545 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b8d31f2[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:32,545 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:32,545 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:32,546 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:32,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:32,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:32,546 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:32,546 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:32,547 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:32,547 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:32,547 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:32,548 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:32,548 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:32,548 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:32,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:32,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:32,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:32,548 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:32,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:32,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:32,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:32,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:32,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:32,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:32,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:32,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:32,551 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:32,551 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:32,551 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:32,551 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:32,551 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:32,551 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:32,552 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:32,552 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:32,552 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:32,552 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:32,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:32,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:32,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:32,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:32,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:32,553 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:32,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:32,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:32,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:32,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:32,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:32,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:32,554 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:32,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:32,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:32,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:32,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:32,560 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:32,560 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:32,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:32,560 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:32,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:32,560 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:32,560 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:32,560 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:32,561 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:32,561 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:32,561 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:32,561 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:32,561 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:32,561 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:32,561 DEBUG [Thread-2498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:32,561 DEBUG [Thread-2498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:32,561 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:55:32,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:32,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:32,661 DEBUG [Thread-2498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:32,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:32,663 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:32,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:32,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:32,664 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:32,664 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:32,664 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:32,664 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:32,665 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:32,665 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:32,667 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:32,667 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:32,667 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:32,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:32,667 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:32,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:32,667 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:32,667 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:32,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:32,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:32,668 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:32,668 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:32,668 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:32,668 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:32,668 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:32,668 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:32,668 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:32,668 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:32,668 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:32,669 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. started... 2023-03-29 03:55:32,669 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. as already flushing 2023-03-29 03:55:32,669 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:32,669 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:32,669 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:32,669 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:32,669 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:32,669 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:32,670 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:32,670 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:32,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:32,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:32,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:32,671 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:32,671 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:32,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:32,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:32,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:32,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:32,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:32,672 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:32,672 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16eb2ace[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:32,672 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:32,672 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:32,673 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:32,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:32,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:32,673 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:32,673 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:32,673 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:32,673 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:32,673 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:32,676 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:32,676 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:32,676 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:32,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:32,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:32,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:32,676 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:32,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:32,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:32,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:32,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:32,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:32,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:32,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:32,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:32,679 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:32,679 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:32,679 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:32,679 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:32,679 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:32,679 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:32,679 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:32,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51970 deadline: 1680062192674, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:32,679 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:32,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51990 deadline: 1680062192677, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:32,680 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:32,681 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:32,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:32,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:32,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:32,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:32,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:32,681 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:32,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:32,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:32,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:32,681 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:32,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:32,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:32,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:32,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:32,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:32,682 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:32,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:32,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:32,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:32,684 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:32,684 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:32,684 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:32,685 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:32,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:32,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:32,685 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:32,685 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:32,685 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:55:32,685 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:32,685 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:32,685 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:32,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:32,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:32,685 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:32,685 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:32,685 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:32,686 DEBUG [Thread-2498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:32,686 DEBUG [Thread-2498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:32,743 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=290 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/B/d239385552404f1c8762ef7d80ec858c 2023-03-29 03:55:32,756 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=290 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/C/ee69bc58c4634710ac8e3827656bfc6b 2023-03-29 03:55:32,761 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/A/8cf0f8e20dd94b87b97107cb27a9a2e9 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/8cf0f8e20dd94b87b97107cb27a9a2e9 2023-03-29 03:55:32,766 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/8cf0f8e20dd94b87b97107cb27a9a2e9, entries=250, sequenceid=290, filesize=39.2 K 2023-03-29 03:55:32,767 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/B/d239385552404f1c8762ef7d80ec858c as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/d239385552404f1c8762ef7d80ec858c 2023-03-29 03:55:32,770 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/d239385552404f1c8762ef7d80ec858c, entries=150, sequenceid=290, filesize=11.9 K 2023-03-29 03:55:32,770 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/C/ee69bc58c4634710ac8e3827656bfc6b as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/ee69bc58c4634710ac8e3827656bfc6b 2023-03-29 03:55:32,773 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/ee69bc58c4634710ac8e3827656bfc6b, entries=150, sequenceid=290, filesize=11.9 K 2023-03-29 03:55:32,774 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~360 KB/368640, currentSize=60.38 KB/61830 for a5c78075cc0f4fd1864c2f0339ad3d67 in 1282ms, sequenceid=290, compaction requested=true 2023-03-29 03:55:32,774 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a5c78075cc0f4fd1864c2f0339ad3d67: 2023-03-29 03:55:32,774 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:55:32,774 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-29 03:55:32,774 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-03-29 03:55:32,774 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:55:32,774 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:55:32,775 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-03-29 03:55:32,775 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-03-29 03:55:32,775 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. because compaction request was cancelled 2023-03-29 03:55:32,775 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-03-29 03:55:32,775 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 139502 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-29 03:55:32,775 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1912): a5c78075cc0f4fd1864c2f0339ad3d67/A is initiating minor compaction (all files) 2023-03-29 03:55:32,775 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a5c78075cc0f4fd1864c2f0339ad3d67/A in TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:32,775 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/d736a34884574dada571d2b9d7c1b07c, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/cf6aa67f579649e7b0fe9f2277172a14, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/5f65fa1579ec4cc6894a148c8d3fa060, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/8cf0f8e20dd94b87b97107cb27a9a2e9] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp, totalSize=136.2 K 2023-03-29 03:55:32,775 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-03-29 03:55:32,775 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-03-29 03:55:32,775 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. because compaction request was cancelled 2023-03-29 03:55:32,776 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting d736a34884574dada571d2b9d7c1b07c, keycount=150, bloomtype=ROW, size=25.7 K, encoding=NONE, compression=NONE, seqNum=218, earliestPutTs=1680062129043 2023-03-29 03:55:32,776 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting cf6aa67f579649e7b0fe9f2277172a14, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=255, earliestPutTs=1680062129145 2023-03-29 03:55:32,776 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 5f65fa1579ec4cc6894a148c8d3fa060, keycount=300, bloomtype=ROW, size=46.1 K, encoding=NONE, compression=NONE, seqNum=269, earliestPutTs=1680062131363 2023-03-29 03:55:32,776 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 8cf0f8e20dd94b87b97107cb27a9a2e9, keycount=250, bloomtype=ROW, size=39.2 K, encoding=NONE, compression=NONE, seqNum=290, earliestPutTs=1680062131467 2023-03-29 03:55:32,785 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] throttle.PressureAwareThroughputController(145): a5c78075cc0f4fd1864c2f0339ad3d67#A#compaction#390 average throughput is 4.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:55:32,786 DEBUG [Thread-2498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:32,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:32,788 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:32,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:32,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:32,796 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:32,796 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:32,796 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:32,796 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:32,796 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:32,796 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:32,801 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:32,801 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:32,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:32,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:32,801 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:32,802 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:32,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:32,802 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:32,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:32,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:32,802 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:32,802 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:32,802 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:32,802 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:32,803 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:32,803 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:32,804 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:32,804 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:32,804 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:32,804 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. started... 2023-03-29 03:55:32,804 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a5c78075cc0f4fd1864c2f0339ad3d67 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-29 03:55:32,804 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5c78075cc0f4fd1864c2f0339ad3d67, store=A 2023-03-29 03:55:32,804 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:55:32,804 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5c78075cc0f4fd1864c2f0339ad3d67, store=B 2023-03-29 03:55:32,804 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:55:32,805 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5c78075cc0f4fd1864c2f0339ad3d67, store=C 2023-03-29 03:55:32,805 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:55:33,215 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/A/56dfba20777f471b83b8bb998260a06c as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/56dfba20777f471b83b8bb998260a06c 2023-03-29 03:55:33,220 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in a5c78075cc0f4fd1864c2f0339ad3d67/A of a5c78075cc0f4fd1864c2f0339ad3d67 into 56dfba20777f471b83b8bb998260a06c(size=26.0 K), total size for store is 26.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:55:33,220 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a5c78075cc0f4fd1864c2f0339ad3d67: 2023-03-29 03:55:33,220 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67., storeName=a5c78075cc0f4fd1864c2f0339ad3d67/A, priority=12, startTime=1680062132774; duration=0sec 2023-03-29 03:55:33,221 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:55:33,229 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303292d3df0d78b1847d58b0d5e3c9587a7fe to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303292d3df0d78b1847d58b0d5e3c9587a7fe 2023-03-29 03:55:33,236 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=305, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/A/3c797965921e42cca36338c3c3f382af 2023-03-29 03:55:33,648 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=305 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/B/2aa68d67535d414bb5f9875f7bf90a93 2023-03-29 03:55:33,671 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=305 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/C/08b2669d45bd4ab9b031f7107d833fbb 2023-03-29 03:55:33,676 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/A/3c797965921e42cca36338c3c3f382af as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/3c797965921e42cca36338c3c3f382af 2023-03-29 03:55:33,679 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/3c797965921e42cca36338c3c3f382af, entries=150, sequenceid=305, filesize=25.4 K 2023-03-29 03:55:33,680 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/B/2aa68d67535d414bb5f9875f7bf90a93 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/2aa68d67535d414bb5f9875f7bf90a93 2023-03-29 03:55:33,683 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/2aa68d67535d414bb5f9875f7bf90a93, entries=150, sequenceid=305, filesize=11.9 K 2023-03-29 03:55:33,684 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/C/08b2669d45bd4ab9b031f7107d833fbb as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/08b2669d45bd4ab9b031f7107d833fbb 2023-03-29 03:55:33,687 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/08b2669d45bd4ab9b031f7107d833fbb, entries=150, sequenceid=305, filesize=11.9 K 2023-03-29 03:55:33,688 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=0 B/0 for a5c78075cc0f4fd1864c2f0339ad3d67 in 884ms, sequenceid=305, compaction requested=true 2023-03-29 03:55:33,688 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a5c78075cc0f4fd1864c2f0339ad3d67: 2023-03-29 03:55:33,688 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:33,688 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:33,688 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:33,688 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:33,688 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:33,688 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:33,690 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:33,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:33,690 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:33,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:33,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:33,690 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:33,690 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:33,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:33,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:33,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:33,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:33,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:33,692 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:33,692 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@26b97ceb[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:33,692 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:33,692 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:33,694 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:33,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:33,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:33,694 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:33,694 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:33,694 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:33,694 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:33,694 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:33,696 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:33,696 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:33,696 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:33,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:33,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:33,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:33,696 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:33,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:33,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:33,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:33,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:33,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:33,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:33,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:33,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:33,698 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:33,698 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:33,698 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:33,698 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:33,698 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:33,698 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:33,702 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:33,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:33,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:33,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:33,702 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:33,702 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:33,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:33,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:33,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:33,703 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:33,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:33,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:33,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:33,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:33,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:33,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:33,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:33,704 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:33,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:33,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:33,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:33,707 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:33,707 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:33,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:33,707 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:33,707 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:33,707 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:33,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:33,707 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:33,707 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:33,707 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:33,707 DEBUG [Thread-2498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:33,707 DEBUG [Thread-2498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:33,707 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:33,707 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:33,708 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:33,708 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:33,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:33,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:33,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on a5c78075cc0f4fd1864c2f0339ad3d67 2023-03-29 03:55:33,802 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a5c78075cc0f4fd1864c2f0339ad3d67 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-29 03:55:33,803 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5c78075cc0f4fd1864c2f0339ad3d67, store=A 2023-03-29 03:55:33,803 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:55:33,803 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5c78075cc0f4fd1864c2f0339ad3d67, store=B 2023-03-29 03:55:33,803 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:55:33,803 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5c78075cc0f4fd1864c2f0339ad3d67, store=C 2023-03-29 03:55:33,803 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:55:33,807 DEBUG [Thread-2498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:33,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:33,811 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:33,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:33,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:33,813 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:33,813 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:33,814 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:33,814 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:33,814 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:33,814 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:33,815 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:33,815 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:33,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:33,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:33,816 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:33,816 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:33,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:33,816 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:33,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:33,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:33,817 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:33,817 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:33,817 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:33,817 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:33,817 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:33,817 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:33,817 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:33,817 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:33,817 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:33,817 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. started... 2023-03-29 03:55:33,817 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. as already flushing 2023-03-29 03:55:33,817 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:33,817 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:33,818 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:33,818 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:33,818 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:33,818 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:33,820 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:33,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:33,820 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:33,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:33,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:33,820 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:33,820 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:33,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:33,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:33,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:33,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:33,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:33,822 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:33,822 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a0b997f[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:33,822 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:33,822 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:33,824 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:33,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:33,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:33,824 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:33,824 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:33,824 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:33,824 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:33,824 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:33,825 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:33,825 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:33,825 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:33,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:33,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:33,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:33,825 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:33,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:33,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:33,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:33,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:33,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:33,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:33,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:33,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:33,827 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:33,827 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:33,827 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:33,827 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:33,827 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:33,828 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:33,830 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:33,830 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:33,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:33,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:33,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:33,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:33,830 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:33,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:33,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:33,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:33,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:33,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:33,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:33,831 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:33,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:33,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:33,832 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:33,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:33,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:33,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:33,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:33,835 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:33,835 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:33,835 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:33,835 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:33,835 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:33,835 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:33,835 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:33,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:33,835 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:33,835 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:33,835 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:33,835 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:33,835 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:33,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:33,836 DEBUG [Thread-2498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:33,836 DEBUG [Thread-2498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:33,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:33,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:33,837 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:55:33,865 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680062133848, max timestamp=1680062133863Num uniques 859; 2023-03-29 03:55:33,865 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680062133848, max timestamp=1680062133863Num uniques 859; 2023-03-29 03:55:33,866 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-29 03:55:33,866 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-29 03:55:33,908 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:33,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51970 deadline: 1680062193898, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:33,910 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:33,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51990 deadline: 1680062193898, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:33,936 DEBUG [Thread-2498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:33,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:33,938 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:33,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:33,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:33,939 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:33,939 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:33,939 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:33,939 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:33,939 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:33,939 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:33,940 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:33,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:33,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:33,941 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:33,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:33,941 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:33,941 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:33,941 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:33,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:33,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:33,942 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:33,942 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:33,942 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:33,942 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:33,942 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:33,942 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:33,942 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:33,942 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:33,942 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:33,942 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. started... 2023-03-29 03:55:33,942 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. as already flushing 2023-03-29 03:55:33,942 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:33,942 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:33,942 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:33,942 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:33,942 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:33,942 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:33,944 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:33,944 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:33,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:33,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:33,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:33,944 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:33,944 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:33,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:33,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:33,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:33,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:33,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:33,945 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:33,945 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@689b256d[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:33,945 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:33,946 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:33,947 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:33,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:33,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:33,948 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:33,948 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:33,948 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:33,948 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:33,948 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:33,949 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:33,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:33,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:33,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:33,949 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:33,949 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:33,949 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:33,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:33,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:33,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:33,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:33,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:33,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:33,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:33,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:33,953 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:33,953 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:33,953 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:33,953 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:33,953 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:33,953 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:33,954 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:33,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:33,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:33,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:33,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:33,955 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:33,955 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:33,955 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:33,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:33,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:33,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:33,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:33,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:33,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:33,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:33,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:33,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:33,956 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:33,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:33,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:33,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:33,958 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:33,958 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:33,958 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:33,958 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:33,958 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:33,958 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:33,958 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:33,958 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:33,959 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:33,959 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:33,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:33,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:33,959 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:33,959 DEBUG [Thread-2498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:33,959 DEBUG [Thread-2498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:33,959 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:33,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:33,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:34,016 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:34,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51970 deadline: 1680062194010, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:34,016 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:34,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51990 deadline: 1680062194011, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:34,059 DEBUG [Thread-2498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:34,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:34,061 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:34,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:34,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:34,062 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:34,062 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:34,062 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:34,062 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:34,063 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:34,063 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:34,066 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:34,066 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:34,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:34,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:34,066 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:34,066 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:34,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:34,066 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:34,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:34,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:34,067 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:34,067 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:34,067 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:34,067 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:34,067 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:34,067 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:34,067 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:34,067 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:34,067 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:34,067 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. started... 2023-03-29 03:55:34,067 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. as already flushing 2023-03-29 03:55:34,068 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:34,068 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:34,068 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:34,068 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:34,068 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:34,068 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:34,069 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:34,069 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:34,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:34,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:34,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:34,069 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:34,069 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:34,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:34,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:34,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:34,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:34,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:34,071 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:34,071 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2769cfea[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:34,071 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:34,071 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:34,072 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:34,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:34,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:34,072 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:34,072 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:34,072 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:34,072 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:34,072 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:34,074 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:34,074 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:34,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:34,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:34,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:34,075 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:34,075 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:34,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:34,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:34,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:34,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:34,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:34,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:34,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:34,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:34,077 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:34,077 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:34,078 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:34,078 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:34,078 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:34,078 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:34,080 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:34,080 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:34,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:34,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:34,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:34,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:34,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:34,080 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:34,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:34,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:34,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:34,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:34,081 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:34,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:34,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:34,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:34,082 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:34,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:34,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:34,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:34,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:34,087 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:34,087 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:34,087 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:34,087 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:34,087 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:34,087 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:34,087 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:34,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:34,088 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:34,088 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:34,088 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:34,088 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:55:34,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:34,088 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:34,088 DEBUG [Thread-2498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:34,088 DEBUG [Thread-2498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:34,089 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:34,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:34,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:34,188 DEBUG [Thread-2498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:34,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:34,190 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:34,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:34,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:34,192 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:34,192 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:34,192 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:34,192 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:34,192 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:34,192 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:34,193 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:34,193 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:34,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:34,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:34,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:34,194 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:34,194 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:34,194 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:34,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:34,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:34,195 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:34,195 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:34,195 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:34,195 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:34,195 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:34,195 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:34,196 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:34,196 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:34,196 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:34,196 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. started... 2023-03-29 03:55:34,196 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. as already flushing 2023-03-29 03:55:34,196 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:34,196 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:34,196 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:34,196 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:34,196 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:34,196 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:34,198 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:34,198 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:34,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:34,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:34,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:34,198 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:34,198 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:34,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:34,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:34,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:34,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:34,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:34,199 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:34,199 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4adab1c8[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:34,199 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:34,200 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:34,202 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:34,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:34,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:34,202 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:34,202 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:34,202 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:34,203 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:34,203 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:34,204 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:34,204 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:34,204 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:34,205 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:34,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:34,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:34,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:34,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:34,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:34,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:34,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:34,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:34,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:34,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:34,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:34,207 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:34,207 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:34,207 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:34,207 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:34,207 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:34,207 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:34,209 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:34,209 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:34,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:34,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:34,209 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:34,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:34,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:34,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:34,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:34,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:34,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:34,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:34,210 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:34,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:34,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:34,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:34,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:34,211 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:34,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:34,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:34,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:34,213 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:34,213 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:34,213 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:34,213 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:34,213 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:34,213 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:34,213 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:34,213 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:34,213 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:34,213 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:34,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:34,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:34,213 DEBUG [Thread-2498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:34,214 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:34,214 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:34,214 DEBUG [Thread-2498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:34,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:34,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:34,217 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032903b1f440c7874353a58fd17b635513ee to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032903b1f440c7874353a58fd17b635513ee 2023-03-29 03:55:34,220 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:34,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51970 deadline: 1680062194217, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:34,220 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:34,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51990 deadline: 1680062194217, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:34,314 DEBUG [Thread-2498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:34,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:34,316 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:34,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:34,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:34,317 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:34,317 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:34,317 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:34,317 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:34,318 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:34,318 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:34,320 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:34,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:34,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:34,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:34,321 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:34,321 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:34,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:34,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:34,322 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:34,322 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:34,322 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:34,322 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:34,322 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:34,322 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:34,326 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:34,326 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:34,328 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:34,328 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:34,328 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:34,328 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. started... 2023-03-29 03:55:34,328 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. as already flushing 2023-03-29 03:55:34,328 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:34,328 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:34,328 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:34,328 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:34,328 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:34,328 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:34,330 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:34,330 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:34,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:34,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:34,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:34,330 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:34,330 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:34,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:34,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:34,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:34,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:34,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:34,331 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:34,332 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6cbc098d[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:34,332 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:34,332 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:34,333 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:34,333 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:34,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:34,333 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:34,333 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:34,333 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:34,333 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:34,333 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:34,335 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:34,335 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:34,335 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:34,335 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:34,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:34,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:34,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:34,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:34,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:34,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:34,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:34,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:34,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:34,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:34,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:34,338 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:34,338 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:34,338 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:34,338 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:34,338 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:34,339 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:34,340 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:34,341 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:34,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:34,341 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:34,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:34,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:34,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:34,341 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:34,341 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:34,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:34,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:34,341 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:34,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:34,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:34,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:34,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:34,342 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:34,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:34,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:34,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:34,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:34,345 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:34,345 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:34,345 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:34,345 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:34,345 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:34,345 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:34,345 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:55:34,345 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:34,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:34,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:34,345 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:34,346 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:34,346 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:34,346 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:34,346 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:34,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:34,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:34,346 DEBUG [Thread-2498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:34,346 DEBUG [Thread-2498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:34,446 DEBUG [Thread-2498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:34,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:34,449 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:34,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:34,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:34,452 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:34,452 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:34,452 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:34,452 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:34,453 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:34,453 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:34,456 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:34,456 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:34,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:34,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:34,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:34,457 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:34,457 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:34,458 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:34,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:34,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:34,458 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:34,478 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:34,479 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:34,479 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:34,479 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:34,479 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:34,479 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:34,479 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:34,479 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:34,479 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. started... 2023-03-29 03:55:34,479 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. as already flushing 2023-03-29 03:55:34,479 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:34,479 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:34,479 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:34,479 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:34,479 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:34,479 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:34,483 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:34,483 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:34,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:34,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:34,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:34,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:34,483 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:34,483 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:34,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:34,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:34,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:34,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:34,485 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:34,485 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5dbdf93a[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:34,485 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:34,485 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:34,486 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:34,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:34,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:34,486 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:34,487 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:34,487 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:34,487 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:34,487 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:34,489 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:34,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:34,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:34,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:34,490 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:34,490 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:34,490 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:34,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:34,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:34,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:34,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:34,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:34,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:34,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:34,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:34,494 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:34,494 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:34,494 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:34,494 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:34,494 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:34,494 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:34,498 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:34,498 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:34,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:34,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:34,499 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:34,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:34,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:34,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:34,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:34,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:34,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:34,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:34,502 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:34,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:34,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:34,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:34,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:34,503 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:34,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:34,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:34,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:34,513 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:34,513 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:34,513 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:34,513 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:34,513 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:34,513 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:34,513 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:34,513 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:34,513 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:34,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:34,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:34,514 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:34,514 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:34,514 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:34,514 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:55:34,514 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:34,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:34,514 DEBUG [Thread-2498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:34,514 DEBUG [Thread-2498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:34,525 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:34,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51970 deadline: 1680062194521, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:34,525 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:34,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51990 deadline: 1680062194521, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:34,614 DEBUG [Thread-2498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:34,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:34,616 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:34,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:34,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:34,618 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:34,618 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:34,618 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:34,618 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:34,618 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:34,618 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:34,619 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:34,619 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:34,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:34,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:34,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:34,620 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:34,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:34,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:34,624 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:34,624 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:34,624 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:34,625 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:34,625 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=317, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/A/6ab685bea079460fba8340e942e7f966 2023-03-29 03:55:34,625 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:34,625 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:34,625 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:34,624 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:34,626 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:34,626 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:34,627 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:34,627 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. started... 2023-03-29 03:55:34,627 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. as already flushing 2023-03-29 03:55:34,628 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:34,628 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:34,628 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:34,628 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:34,628 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:34,629 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:34,633 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:34,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:34,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:34,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:34,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:34,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:34,634 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:34,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:34,635 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:34,635 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:34,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:34,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:34,637 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:34,637 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@34c26bf8[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:34,637 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:34,637 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:34,638 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:34,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:34,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:34,638 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:34,639 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:34,639 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:34,639 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:34,639 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:34,640 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=317 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/B/7ec4b24c6ebd41f2b2703f5cc530b77e 2023-03-29 03:55:34,641 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:34,641 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:34,641 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:34,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:34,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:34,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:34,641 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:34,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:34,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:34,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:34,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:34,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:34,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:34,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:34,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:34,646 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:34,646 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:34,646 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:34,646 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:34,646 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:34,646 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:34,651 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:34,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:34,651 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:34,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:34,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:34,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:34,651 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:34,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:34,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:34,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:34,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:34,652 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:34,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:34,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:34,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:34,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:34,656 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:34,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:34,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:34,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:34,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:34,659 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:34,659 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:34,659 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:34,659 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:34,659 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:34,659 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:34,659 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:34,659 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:34,659 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:34,659 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:34,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:34,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:34,659 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:34,660 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:34,660 DEBUG [Thread-2498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:34,660 DEBUG [Thread-2498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:34,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:34,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:34,660 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:55:34,760 DEBUG [Thread-2498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:34,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:34,762 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:34,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:34,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:34,763 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:34,763 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:34,763 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:34,763 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:34,763 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:34,763 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:34,764 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:34,765 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:34,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:34,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:34,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:34,765 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:34,765 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:34,765 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:34,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:34,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:34,766 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:34,766 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:34,766 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:34,766 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:34,766 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:34,766 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:34,766 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:34,766 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:34,767 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. started... 2023-03-29 03:55:34,767 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. as already flushing 2023-03-29 03:55:34,767 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:34,767 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:34,767 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:34,767 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:34,767 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:34,767 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:34,768 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:34,768 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:34,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:34,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:34,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:34,769 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:34,769 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:34,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:34,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:34,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:34,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:34,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:34,770 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:34,770 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:34,770 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:34,770 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b89d859[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:34,772 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:34,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:34,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:34,773 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:34,773 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:34,773 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:34,773 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:34,773 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:34,774 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:34,774 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:34,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:34,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:34,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:34,774 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:34,774 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:34,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:34,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:34,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:34,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:34,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:34,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:34,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:34,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:34,776 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:34,776 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:34,776 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:34,776 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:34,776 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:34,776 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:34,778 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:34,778 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:34,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:34,778 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:34,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:34,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:34,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:34,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:34,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:34,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:34,779 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:34,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:34,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:34,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:34,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:34,780 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:34,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:34,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:34,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:34,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:34,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:34,782 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:34,782 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:34,782 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:34,782 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:34,782 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:34,782 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:34,782 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:34,782 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:34,782 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:34,782 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:34,782 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:34,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:34,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:34,782 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:34,782 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:55:34,782 DEBUG [Thread-2498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:34,782 DEBUG [Thread-2498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:34,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:34,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:34,883 DEBUG [Thread-2498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:34,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:34,885 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:34,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:34,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:34,886 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:34,886 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:34,886 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:34,886 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:34,887 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:34,887 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:34,889 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:34,889 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:34,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:34,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:34,889 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:34,889 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:34,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:34,890 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:34,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:34,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:34,890 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:34,890 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:34,891 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:34,891 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:34,891 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:34,891 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:34,891 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:34,891 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:34,891 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:34,891 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. started... 2023-03-29 03:55:34,891 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. as already flushing 2023-03-29 03:55:34,892 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:34,892 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:34,892 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:34,892 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:34,892 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:34,892 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:34,893 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:34,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:34,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:34,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:34,893 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:34,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:34,894 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:34,894 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:34,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:34,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:34,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:34,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:34,896 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:34,896 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@27f55cc4[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:34,896 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:34,896 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:34,898 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:34,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:34,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:34,898 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:34,898 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:34,898 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:34,898 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:34,898 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:34,899 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:34,899 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:34,900 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:34,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:34,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:34,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:34,900 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:34,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:34,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:34,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:34,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:34,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:34,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:34,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:34,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:34,902 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:34,902 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:34,902 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:34,903 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:34,903 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:34,903 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:34,908 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:34,908 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:34,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:34,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:34,908 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:34,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:34,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:34,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:34,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:34,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:34,909 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:34,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:34,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:34,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:34,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:34,910 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:34,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:34,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:34,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:34,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:34,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:34,913 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:34,913 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:34,913 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:34,914 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:34,913 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:34,914 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:34,913 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:34,914 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:34,914 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:34,913 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:34,914 DEBUG [Thread-2498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:34,914 DEBUG [Thread-2498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:34,914 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:34,914 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:34,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:34,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:34,914 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:55:34,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:34,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:35,014 DEBUG [Thread-2498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:35,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:35,017 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:35,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:35,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:35,018 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:35,018 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:35,018 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:35,018 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:35,019 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:35,019 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:35,020 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:35,020 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:35,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:35,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:35,020 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:35,020 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:35,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:35,022 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:35,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:35,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:35,023 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:35,024 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:35,024 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:35,024 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:35,024 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:35,024 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:35,026 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:35,026 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:35,027 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:35,027 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. started... 2023-03-29 03:55:35,027 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. as already flushing 2023-03-29 03:55:35,027 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:35,027 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:35,027 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:35,027 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:35,027 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:35,027 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:35,031 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:35,031 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:35,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:35,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:35,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:35,031 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:35,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:35,032 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:35,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:35,032 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:35,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:35,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51970 deadline: 1680062195027, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:35,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:35,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:35,033 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:35,033 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@15b30529[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:35,033 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:35,033 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:35,034 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:35,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:35,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:35,035 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:35,035 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:35,035 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:35,035 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:35,035 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:35,036 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:35,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51990 deadline: 1680062195030, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:35,036 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:35,036 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:35,036 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:35,042 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:35,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:35,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:35,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:35,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:35,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:35,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:35,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:35,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:35,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:35,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:35,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:35,046 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:35,046 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:35,046 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:35,047 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:35,047 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:35,047 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:35,052 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:35,052 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:35,052 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:35,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:35,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:35,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:35,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:35,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:35,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:35,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:35,053 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:35,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:35,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:35,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:35,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:35,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:35,054 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:35,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:35,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:35,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:35,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:35,056 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:35,056 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:35,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:35,057 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:35,057 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:35,056 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:35,056 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:35,057 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:35,057 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:35,057 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:35,057 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:35,057 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:35,057 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:35,057 DEBUG [Thread-2498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:35,057 DEBUG [Thread-2498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:35,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:35,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:35,058 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:55:35,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:35,073 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=317 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/C/72d73d4d55da4fd0a1bddb1f7fbf7dd9 2023-03-29 03:55:35,081 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/A/6ab685bea079460fba8340e942e7f966 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/6ab685bea079460fba8340e942e7f966 2023-03-29 03:55:35,087 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/6ab685bea079460fba8340e942e7f966, entries=300, sequenceid=317, filesize=46.1 K 2023-03-29 03:55:35,092 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/B/7ec4b24c6ebd41f2b2703f5cc530b77e as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/7ec4b24c6ebd41f2b2703f5cc530b77e 2023-03-29 03:55:35,097 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/7ec4b24c6ebd41f2b2703f5cc530b77e, entries=150, sequenceid=317, filesize=11.9 K 2023-03-29 03:55:35,097 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/C/72d73d4d55da4fd0a1bddb1f7fbf7dd9 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/72d73d4d55da4fd0a1bddb1f7fbf7dd9 2023-03-29 03:55:35,101 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/72d73d4d55da4fd0a1bddb1f7fbf7dd9, entries=150, sequenceid=317, filesize=11.9 K 2023-03-29 03:55:35,102 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=125.23 KB/128240 for a5c78075cc0f4fd1864c2f0339ad3d67 in 1300ms, sequenceid=317, compaction requested=true 2023-03-29 03:55:35,102 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a5c78075cc0f4fd1864c2f0339ad3d67: 2023-03-29 03:55:35,102 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:55:35,102 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-29 03:55:35,103 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:55:35,103 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-29 03:55:35,103 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:55:35,104 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 99808 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-29 03:55:35,104 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1912): a5c78075cc0f4fd1864c2f0339ad3d67/A is initiating minor compaction (all files) 2023-03-29 03:55:35,104 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a5c78075cc0f4fd1864c2f0339ad3d67/A in TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:35,104 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/56dfba20777f471b83b8bb998260a06c, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/3c797965921e42cca36338c3c3f382af, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/6ab685bea079460fba8340e942e7f966] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp, totalSize=97.5 K 2023-03-29 03:55:35,104 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 56dfba20777f471b83b8bb998260a06c, keycount=150, bloomtype=ROW, size=26.0 K, encoding=NONE, compression=NONE, seqNum=290, earliestPutTs=1680062131467 2023-03-29 03:55:35,105 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49256 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-29 03:55:35,105 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1912): a5c78075cc0f4fd1864c2f0339ad3d67/B is initiating minor compaction (all files) 2023-03-29 03:55:35,105 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a5c78075cc0f4fd1864c2f0339ad3d67/B in TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:35,105 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/c5866dee61e44f8f914008097aa0e9e1, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/d239385552404f1c8762ef7d80ec858c, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/2aa68d67535d414bb5f9875f7bf90a93, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/7ec4b24c6ebd41f2b2703f5cc530b77e] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp, totalSize=48.1 K 2023-03-29 03:55:35,105 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 3c797965921e42cca36338c3c3f382af, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=305, earliestPutTs=1680062131509 2023-03-29 03:55:35,105 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting c5866dee61e44f8f914008097aa0e9e1, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=269, earliestPutTs=1680062131398 2023-03-29 03:55:35,106 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 6ab685bea079460fba8340e942e7f966, keycount=300, bloomtype=ROW, size=46.1 K, encoding=NONE, compression=NONE, seqNum=317, earliestPutTs=1680062133769 2023-03-29 03:55:35,106 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting d239385552404f1c8762ef7d80ec858c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=290, earliestPutTs=1680062131467 2023-03-29 03:55:35,106 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 2aa68d67535d414bb5f9875f7bf90a93, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=305, earliestPutTs=1680062131509 2023-03-29 03:55:35,109 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 7ec4b24c6ebd41f2b2703f5cc530b77e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=317, earliestPutTs=1680062133791 2023-03-29 03:55:35,122 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] throttle.PressureAwareThroughputController(145): a5c78075cc0f4fd1864c2f0339ad3d67#B#compaction#398 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:55:35,125 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] throttle.PressureAwareThroughputController(145): a5c78075cc0f4fd1864c2f0339ad3d67#A#compaction#397 average throughput is 2.19 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:55:35,157 DEBUG [Thread-2498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:35,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:35,159 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:35,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:35,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:35,161 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:35,162 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:35,162 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:35,162 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:35,162 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:35,162 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:35,165 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:35,165 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:35,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:35,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:35,166 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:35,166 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:35,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:35,166 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:35,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:35,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:35,167 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:35,168 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:35,168 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:35,168 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:35,168 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:35,168 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:35,169 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:35,169 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:35,169 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:35,169 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. started... 2023-03-29 03:55:35,169 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a5c78075cc0f4fd1864c2f0339ad3d67 3/3 column families, dataSize=125.23 KB heapSize=363.39 KB 2023-03-29 03:55:35,169 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5c78075cc0f4fd1864c2f0339ad3d67, store=A 2023-03-29 03:55:35,169 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-29 03:55:35,169 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5c78075cc0f4fd1864c2f0339ad3d67, store=B 2023-03-29 03:55:35,169 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-29 03:55:35,170 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5c78075cc0f4fd1864c2f0339ad3d67, store=C 2023-03-29 03:55:35,170 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-29 03:55:35,551 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/B/7e94db7fe04348f6b81bc03a9e88ff21 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/7e94db7fe04348f6b81bc03a9e88ff21 2023-03-29 03:55:35,559 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in a5c78075cc0f4fd1864c2f0339ad3d67/B of a5c78075cc0f4fd1864c2f0339ad3d67 into 7e94db7fe04348f6b81bc03a9e88ff21(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:55:35,559 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a5c78075cc0f4fd1864c2f0339ad3d67: 2023-03-29 03:55:35,559 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67., storeName=a5c78075cc0f4fd1864c2f0339ad3d67/B, priority=12, startTime=1680062135102; duration=0sec 2023-03-29 03:55:35,559 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:55:35,559 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-29 03:55:35,560 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49256 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-29 03:55:35,561 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1912): a5c78075cc0f4fd1864c2f0339ad3d67/C is initiating minor compaction (all files) 2023-03-29 03:55:35,561 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a5c78075cc0f4fd1864c2f0339ad3d67/C in TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:35,561 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/3285c402839d452589d36b939084301f, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/ee69bc58c4634710ac8e3827656bfc6b, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/08b2669d45bd4ab9b031f7107d833fbb, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/72d73d4d55da4fd0a1bddb1f7fbf7dd9] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp, totalSize=48.1 K 2023-03-29 03:55:35,561 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 3285c402839d452589d36b939084301f, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=269, earliestPutTs=1680062131398 2023-03-29 03:55:35,562 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting ee69bc58c4634710ac8e3827656bfc6b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=290, earliestPutTs=1680062131467 2023-03-29 03:55:35,562 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 08b2669d45bd4ab9b031f7107d833fbb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=305, earliestPutTs=1680062131509 2023-03-29 03:55:35,563 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 72d73d4d55da4fd0a1bddb1f7fbf7dd9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=317, earliestPutTs=1680062133791 2023-03-29 03:55:35,569 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] throttle.PressureAwareThroughputController(145): a5c78075cc0f4fd1864c2f0339ad3d67#C#compaction#400 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:55:35,588 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/C/2f3277cda34c4a89a04edc2000b805cd as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/2f3277cda34c4a89a04edc2000b805cd 2023-03-29 03:55:35,590 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230329bb625559ac2e4d17b6ff10f0850b47d1 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230329bb625559ac2e4d17b6ff10f0850b47d1 2023-03-29 03:55:35,594 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in a5c78075cc0f4fd1864c2f0339ad3d67/C of a5c78075cc0f4fd1864c2f0339ad3d67 into 2f3277cda34c4a89a04edc2000b805cd(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:55:35,594 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a5c78075cc0f4fd1864c2f0339ad3d67: 2023-03-29 03:55:35,594 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67., storeName=a5c78075cc0f4fd1864c2f0339ad3d67/C, priority=12, startTime=1680062135103; duration=0sec 2023-03-29 03:55:35,594 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:55:35,597 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=348, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/A/5a9a313901454e239923a9dc8e82b169 2023-03-29 03:55:35,968 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/A/94556f512cbc482dafab99e54df84c91 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/94556f512cbc482dafab99e54df84c91 2023-03-29 03:55:35,972 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a5c78075cc0f4fd1864c2f0339ad3d67/A of a5c78075cc0f4fd1864c2f0339ad3d67 into 94556f512cbc482dafab99e54df84c91(size=26.1 K), total size for store is 26.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:55:35,972 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a5c78075cc0f4fd1864c2f0339ad3d67: 2023-03-29 03:55:35,972 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67., storeName=a5c78075cc0f4fd1864c2f0339ad3d67/A, priority=13, startTime=1680062135102; duration=0sec 2023-03-29 03:55:35,972 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:55:36,022 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=348 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/B/be951641a9b144dfb7cc65cf3d8c8555 2023-03-29 03:55:36,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on a5c78075cc0f4fd1864c2f0339ad3d67 2023-03-29 03:55:36,044 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. as already flushing 2023-03-29 03:55:36,064 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:36,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51970 deadline: 1680062196057, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:36,069 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:36,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51990 deadline: 1680062196064, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:36,168 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:36,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51970 deadline: 1680062196165, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:36,175 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:36,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51990 deadline: 1680062196170, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:36,374 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:36,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51970 deadline: 1680062196369, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:36,380 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:36,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51990 deadline: 1680062196376, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:36,436 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=62.62 KB at sequenceid=348 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/C/a2ceed1c461c495287a0cd19269dd47d 2023-03-29 03:55:36,440 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/A/5a9a313901454e239923a9dc8e82b169 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/5a9a313901454e239923a9dc8e82b169 2023-03-29 03:55:36,444 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/5a9a313901454e239923a9dc8e82b169, entries=150, sequenceid=348, filesize=25.4 K 2023-03-29 03:55:36,445 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/B/be951641a9b144dfb7cc65cf3d8c8555 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/be951641a9b144dfb7cc65cf3d8c8555 2023-03-29 03:55:36,449 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/be951641a9b144dfb7cc65cf3d8c8555, entries=150, sequenceid=348, filesize=11.9 K 2023-03-29 03:55:36,450 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/C/a2ceed1c461c495287a0cd19269dd47d as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/a2ceed1c461c495287a0cd19269dd47d 2023-03-29 03:55:36,463 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/a2ceed1c461c495287a0cd19269dd47d, entries=150, sequenceid=348, filesize=11.9 K 2023-03-29 03:55:36,464 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~125.23 KB/128240, heapSize ~363.34 KB/372064, currentSize=53.67 KB/54960 for a5c78075cc0f4fd1864c2f0339ad3d67 in 1295ms, sequenceid=348, compaction requested=false 2023-03-29 03:55:36,464 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a5c78075cc0f4fd1864c2f0339ad3d67: 2023-03-29 03:55:36,464 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:36,464 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:36,464 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:36,464 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:36,464 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:36,464 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:36,466 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:36,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:36,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:36,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:36,466 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:36,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:36,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:36,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:36,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:36,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:36,468 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:36,468 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@328931cc[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:36,468 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:36,468 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:36,471 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:36,471 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:36,471 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:36,472 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:36,471 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:36,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:36,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:36,472 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:36,472 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:36,472 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:36,473 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:36,473 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:36,473 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:36,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:36,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:36,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:36,473 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:36,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:36,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:36,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:36,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:36,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:36,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:36,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:36,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:36,482 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:36,482 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:36,482 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:36,482 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:36,482 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:36,482 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:36,485 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:36,485 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:36,485 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:36,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:36,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:36,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:36,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:36,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:36,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:36,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:36,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:36,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:36,487 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:36,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:36,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:36,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:36,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:36,491 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:36,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:36,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:36,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:36,495 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:36,495 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:36,495 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:36,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:36,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:36,495 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:36,495 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:36,496 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:36,495 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:36,495 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:36,496 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:36,496 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:36,496 DEBUG [Thread-2498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:36,496 DEBUG [Thread-2498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:36,496 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:36,496 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:36,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:36,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:36,596 DEBUG [Thread-2498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:36,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:36,599 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:36,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:36,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:36,600 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:36,600 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:36,600 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:36,600 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:36,601 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:36,601 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:36,602 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:36,602 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:36,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:36,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:36,602 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:36,602 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:36,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:36,605 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:36,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:36,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:36,606 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:36,609 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:36,609 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:36,609 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:36,609 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:36,609 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:36,610 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:36,610 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:36,611 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:36,611 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. started... 2023-03-29 03:55:36,611 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a5c78075cc0f4fd1864c2f0339ad3d67 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-29 03:55:36,611 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5c78075cc0f4fd1864c2f0339ad3d67, store=A 2023-03-29 03:55:36,611 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:55:36,611 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5c78075cc0f4fd1864c2f0339ad3d67, store=B 2023-03-29 03:55:36,611 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:55:36,611 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5c78075cc0f4fd1864c2f0339ad3d67, store=C 2023-03-29 03:55:36,611 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:55:36,647 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032993ea0204be1343bcb84df175f68f7613 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032993ea0204be1343bcb84df175f68f7613 2023-03-29 03:55:36,653 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=362, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/A/c042bfc59f5c4a119d7c895b8feef858 2023-03-29 03:55:36,679 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=362 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/B/d28acf570c834d2e83f4de9473c9a6c5 2023-03-29 03:55:36,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on a5c78075cc0f4fd1864c2f0339ad3d67 2023-03-29 03:55:36,681 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. as already flushing 2023-03-29 03:55:36,692 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=362 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/C/e0056ddd6eed4fa9981d4d5e16a224aa 2023-03-29 03:55:36,696 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/A/c042bfc59f5c4a119d7c895b8feef858 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/c042bfc59f5c4a119d7c895b8feef858 2023-03-29 03:55:36,700 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/c042bfc59f5c4a119d7c895b8feef858, entries=150, sequenceid=362, filesize=25.4 K 2023-03-29 03:55:36,700 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/B/d28acf570c834d2e83f4de9473c9a6c5 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/d28acf570c834d2e83f4de9473c9a6c5 2023-03-29 03:55:36,704 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/d28acf570c834d2e83f4de9473c9a6c5, entries=150, sequenceid=362, filesize=11.9 K 2023-03-29 03:55:36,705 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/C/e0056ddd6eed4fa9981d4d5e16a224aa as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/e0056ddd6eed4fa9981d4d5e16a224aa 2023-03-29 03:55:36,709 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/e0056ddd6eed4fa9981d4d5e16a224aa, entries=150, sequenceid=362, filesize=11.9 K 2023-03-29 03:55:36,710 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=40.25 KB/41220 for a5c78075cc0f4fd1864c2f0339ad3d67 in 99ms, sequenceid=362, compaction requested=true 2023-03-29 03:55:36,710 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a5c78075cc0f4fd1864c2f0339ad3d67: 2023-03-29 03:55:36,710 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:36,710 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:36,710 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:36,710 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:36,710 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:36,710 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:36,712 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:36,712 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:36,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:36,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:36,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:36,712 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:36,712 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:36,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:36,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:36,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:36,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:36,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:36,714 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:36,714 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@57277cd4[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:36,714 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:36,714 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:36,716 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:36,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:36,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:36,716 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:36,717 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:36,717 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:36,717 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:36,717 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:36,718 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:36,718 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:36,718 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:36,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:36,718 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:36,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:36,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:36,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:36,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:36,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:36,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:36,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:36,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:36,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:36,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:36,721 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:36,721 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:36,721 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:36,721 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:36,721 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:36,721 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:36,722 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:36,722 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:36,722 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:36,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:36,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:36,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:36,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:36,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:36,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:36,723 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:36,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:36,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:36,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:36,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:36,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:36,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:36,726 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:36,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:36,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:36,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:36,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:36,731 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:36,731 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:36,731 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:36,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:36,731 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:36,731 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:36,731 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:36,731 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:36,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:36,731 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:36,732 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:55:36,732 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:36,732 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:36,732 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:36,732 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:36,732 DEBUG [Thread-2498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:36,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:36,732 DEBUG [Thread-2498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:36,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:36,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on a5c78075cc0f4fd1864c2f0339ad3d67 2023-03-29 03:55:36,736 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a5c78075cc0f4fd1864c2f0339ad3d67 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-29 03:55:36,736 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5c78075cc0f4fd1864c2f0339ad3d67, store=A 2023-03-29 03:55:36,736 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:55:36,736 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5c78075cc0f4fd1864c2f0339ad3d67, store=B 2023-03-29 03:55:36,736 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:55:36,736 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5c78075cc0f4fd1864c2f0339ad3d67, store=C 2023-03-29 03:55:36,736 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:55:36,749 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230329a156ea92d05d486ca03c7f37b9e8be2c to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230329a156ea92d05d486ca03c7f37b9e8be2c 2023-03-29 03:55:36,818 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680062136792, max timestamp=1680062136816Num uniques 859; 2023-03-29 03:55:36,818 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-29 03:55:36,818 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680062136792, max timestamp=1680062136816Num uniques 859; 2023-03-29 03:55:36,818 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680062136792, max timestamp=1680062136816Num uniques 859; 2023-03-29 03:55:36,818 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-29 03:55:36,819 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-29 03:55:36,832 DEBUG [Thread-2498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:36,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:36,834 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:36,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:36,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:36,836 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:36,836 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:36,836 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:36,836 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:36,836 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:36,836 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:36,839 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:36,839 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:36,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:36,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:36,839 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:36,839 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:36,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:36,840 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:36,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:36,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:36,840 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:36,841 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:36,841 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:36,841 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:36,841 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:36,841 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:36,841 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:36,841 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:36,841 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:36,841 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. started... 2023-03-29 03:55:36,841 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. as already flushing 2023-03-29 03:55:36,841 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:36,841 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:36,841 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:36,841 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:36,842 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:36,842 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:36,843 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:36,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:36,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:36,843 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:36,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:36,843 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:36,843 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:36,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:36,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:36,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:36,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:36,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:36,845 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:36,845 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:36,845 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ab91988[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:36,845 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:36,846 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:36,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:36,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:36,846 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:36,846 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:36,846 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:36,846 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:36,846 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:36,851 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:36,851 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:36,851 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:36,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:36,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:36,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:36,852 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:36,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:36,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:36,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:36,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:36,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:36,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:36,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:36,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:36,854 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:36,854 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:36,854 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:36,854 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:36,854 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:36,854 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:36,856 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:36,856 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:36,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:36,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:36,856 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:36,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:36,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:36,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:36,856 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:36,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:36,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:36,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:36,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:36,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:36,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:36,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:36,857 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:36,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:36,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:36,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:36,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:36,860 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:36,860 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:36,860 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:36,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:36,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:36,860 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:36,860 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:36,860 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:36,860 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:36,860 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:36,860 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:36,860 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:36,860 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:36,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:36,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:36,860 DEBUG [Thread-2498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:36,860 DEBUG [Thread-2498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:36,860 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:36,861 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:36,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51990 deadline: 1680062196859, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:36,866 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:36,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51970 deadline: 1680062196860, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:36,960 DEBUG [Thread-2498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:36,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:36,962 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:36,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:36,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:36,964 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:36,964 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:36,964 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:36,964 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:36,964 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:36,964 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:36,965 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:36,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51990 deadline: 1680062196962, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:36,967 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:36,967 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:36,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:36,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:36,967 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:36,967 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:36,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:36,968 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:36,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:36,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:36,968 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:36,968 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:36,968 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:36,968 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:36,968 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:36,968 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:36,969 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:36,969 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:36,969 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:36,969 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. started... 2023-03-29 03:55:36,969 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. as already flushing 2023-03-29 03:55:36,969 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:36,969 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:36,969 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:36,969 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:36,969 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:36,969 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:36,971 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:36,971 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:36,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:36,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:36,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:36,971 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:36,971 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:36,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:36,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:36,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:36,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:36,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:36,972 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:36,972 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@57aed5bb[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:36,972 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:36,972 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:36,973 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:36,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51970 deadline: 1680062196967, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:36,973 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:36,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:36,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:36,973 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:36,974 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:36,974 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:36,975 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:36,975 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:36,975 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:36,975 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:36,975 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:36,975 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:36,976 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:36,976 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:36,976 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:36,976 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:36,976 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:36,981 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:36,981 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:36,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:36,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:36,981 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:36,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:36,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:36,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:36,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:36,981 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:36,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:36,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:36,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:36,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:36,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:36,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:36,982 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:36,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:36,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:36,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:36,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:36,985 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:36,985 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:36,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:36,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:36,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:36,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:36,985 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:36,985 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:36,985 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:36,985 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:36,985 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:36,985 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:55:36,985 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:36,986 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:36,985 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:36,986 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:36,986 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:36,986 DEBUG [Thread-2498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:36,986 DEBUG [Thread-2498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:37,086 DEBUG [Thread-2498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:37,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:37,088 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:37,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:37,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:37,089 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:37,089 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:37,089 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:37,089 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:37,090 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:37,090 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:37,091 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:37,091 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:37,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:37,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:37,091 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:37,091 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:37,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:37,091 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:37,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:37,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:37,092 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:37,092 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:37,092 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:37,092 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:37,092 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:37,092 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:37,092 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:37,092 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:37,092 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:37,093 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. started... 2023-03-29 03:55:37,093 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. as already flushing 2023-03-29 03:55:37,093 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:37,093 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:37,093 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:37,093 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:37,093 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:37,093 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:37,094 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:37,094 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:37,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:37,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:37,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:37,095 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:37,095 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:37,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:37,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:37,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:37,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:37,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:37,096 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:37,096 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c5289d3[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:37,096 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:37,096 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:37,103 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:37,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:37,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:37,103 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:37,103 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:37,103 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:37,103 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:37,103 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:37,104 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:37,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:37,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:37,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:37,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:37,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:37,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:37,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:37,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:37,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:37,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:37,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:37,106 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:37,106 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:37,106 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:37,106 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:37,107 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:37,107 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:37,108 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:37,108 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:37,108 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:37,108 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:37,108 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:37,108 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:37,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:37,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:37,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:37,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:37,109 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:37,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:37,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:37,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:37,110 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:37,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:37,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:37,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:37,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:37,112 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:37,112 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:37,112 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:37,112 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:37,112 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:37,112 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:37,112 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:37,112 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:37,112 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:37,112 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:37,112 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:37,112 DEBUG [Thread-2498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:37,113 DEBUG [Thread-2498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:37,113 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:55:37,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:37,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:37,116 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:55:37,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:37,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:37,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:37,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:37,159 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=372, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/A/514dbc28359f4e2abc2a1f9dcd0a3732 2023-03-29 03:55:37,169 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:37,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51990 deadline: 1680062197166, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:37,171 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=372 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/B/a1646ba23916429eb570a4f4da223d52 2023-03-29 03:55:37,176 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:37,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51970 deadline: 1680062197174, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:37,185 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=372 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/C/76d8e08be1494c909ee3994baf678cd9 2023-03-29 03:55:37,188 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/A/514dbc28359f4e2abc2a1f9dcd0a3732 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/514dbc28359f4e2abc2a1f9dcd0a3732 2023-03-29 03:55:37,193 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/514dbc28359f4e2abc2a1f9dcd0a3732, entries=200, sequenceid=372, filesize=32.3 K 2023-03-29 03:55:37,193 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/B/a1646ba23916429eb570a4f4da223d52 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/a1646ba23916429eb570a4f4da223d52 2023-03-29 03:55:37,197 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/a1646ba23916429eb570a4f4da223d52, entries=100, sequenceid=372, filesize=9.5 K 2023-03-29 03:55:37,199 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/C/76d8e08be1494c909ee3994baf678cd9 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/76d8e08be1494c909ee3994baf678cd9 2023-03-29 03:55:37,202 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/76d8e08be1494c909ee3994baf678cd9, entries=100, sequenceid=372, filesize=9.5 K 2023-03-29 03:55:37,202 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for a5c78075cc0f4fd1864c2f0339ad3d67 in 466ms, sequenceid=372, compaction requested=true 2023-03-29 03:55:37,202 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a5c78075cc0f4fd1864c2f0339ad3d67: 2023-03-29 03:55:37,202 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:55:37,202 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:55:37,203 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-29 03:55:37,203 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-29 03:55:37,203 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:55:37,204 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 111868 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-29 03:55:37,204 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1912): a5c78075cc0f4fd1864c2f0339ad3d67/A is initiating minor compaction (all files) 2023-03-29 03:55:37,204 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 46948 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-29 03:55:37,204 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a5c78075cc0f4fd1864c2f0339ad3d67/A in TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:37,204 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1912): a5c78075cc0f4fd1864c2f0339ad3d67/B is initiating minor compaction (all files) 2023-03-29 03:55:37,204 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/94556f512cbc482dafab99e54df84c91, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/5a9a313901454e239923a9dc8e82b169, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/c042bfc59f5c4a119d7c895b8feef858, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/514dbc28359f4e2abc2a1f9dcd0a3732] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp, totalSize=109.2 K 2023-03-29 03:55:37,204 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a5c78075cc0f4fd1864c2f0339ad3d67/B in TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:37,204 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/7e94db7fe04348f6b81bc03a9e88ff21, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/be951641a9b144dfb7cc65cf3d8c8555, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/d28acf570c834d2e83f4de9473c9a6c5, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/a1646ba23916429eb570a4f4da223d52] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp, totalSize=45.8 K 2023-03-29 03:55:37,204 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 94556f512cbc482dafab99e54df84c91, keycount=150, bloomtype=ROW, size=26.1 K, encoding=NONE, compression=NONE, seqNum=317, earliestPutTs=1680062133791 2023-03-29 03:55:37,204 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 7e94db7fe04348f6b81bc03a9e88ff21, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=317, earliestPutTs=1680062133791 2023-03-29 03:55:37,205 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 5a9a313901454e239923a9dc8e82b169, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=348, earliestPutTs=1680062133888 2023-03-29 03:55:37,205 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting be951641a9b144dfb7cc65cf3d8c8555, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=348, earliestPutTs=1680062133888 2023-03-29 03:55:37,205 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting c042bfc59f5c4a119d7c895b8feef858, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=362, earliestPutTs=1680062136055 2023-03-29 03:55:37,205 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting d28acf570c834d2e83f4de9473c9a6c5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=362, earliestPutTs=1680062136055 2023-03-29 03:55:37,205 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 514dbc28359f4e2abc2a1f9dcd0a3732, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=372, earliestPutTs=1680062136685 2023-03-29 03:55:37,206 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting a1646ba23916429eb570a4f4da223d52, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=372, earliestPutTs=1680062136702 2023-03-29 03:55:37,213 DEBUG [Thread-2498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:37,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:37,215 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:37,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:37,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:37,225 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:37,225 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:37,225 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:37,225 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:37,229 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:37,229 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:37,230 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] throttle.PressureAwareThroughputController(145): a5c78075cc0f4fd1864c2f0339ad3d67#B#compaction#410 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:55:37,232 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] throttle.PressureAwareThroughputController(145): a5c78075cc0f4fd1864c2f0339ad3d67#A#compaction#409 average throughput is 0.99 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:55:37,233 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:37,233 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:37,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:37,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:37,233 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:37,233 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:37,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:37,234 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:37,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:37,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:37,234 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:37,240 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:37,240 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:37,240 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:37,240 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:37,240 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:37,243 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:37,243 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:37,243 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:37,243 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. started... 2023-03-29 03:55:37,244 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a5c78075cc0f4fd1864c2f0339ad3d67 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-03-29 03:55:37,244 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5c78075cc0f4fd1864c2f0339ad3d67, store=A 2023-03-29 03:55:37,244 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-29 03:55:37,245 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5c78075cc0f4fd1864c2f0339ad3d67, store=B 2023-03-29 03:55:37,245 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-29 03:55:37,245 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5c78075cc0f4fd1864c2f0339ad3d67, store=C 2023-03-29 03:55:37,245 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-29 03:55:37,252 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/B/d308d6c26d3d436cbd990b94a914c6c7 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/d308d6c26d3d436cbd990b94a914c6c7 2023-03-29 03:55:37,256 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in a5c78075cc0f4fd1864c2f0339ad3d67/B of a5c78075cc0f4fd1864c2f0339ad3d67 into d308d6c26d3d436cbd990b94a914c6c7(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:55:37,256 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a5c78075cc0f4fd1864c2f0339ad3d67: 2023-03-29 03:55:37,256 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67., storeName=a5c78075cc0f4fd1864c2f0339ad3d67/B, priority=12, startTime=1680062137202; duration=0sec 2023-03-29 03:55:37,256 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:55:37,257 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-29 03:55:37,259 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 46948 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-29 03:55:37,259 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1912): a5c78075cc0f4fd1864c2f0339ad3d67/C is initiating minor compaction (all files) 2023-03-29 03:55:37,259 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a5c78075cc0f4fd1864c2f0339ad3d67/C in TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:37,259 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/2f3277cda34c4a89a04edc2000b805cd, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/a2ceed1c461c495287a0cd19269dd47d, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/e0056ddd6eed4fa9981d4d5e16a224aa, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/76d8e08be1494c909ee3994baf678cd9] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp, totalSize=45.8 K 2023-03-29 03:55:37,260 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 2f3277cda34c4a89a04edc2000b805cd, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=317, earliestPutTs=1680062133791 2023-03-29 03:55:37,260 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting a2ceed1c461c495287a0cd19269dd47d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=348, earliestPutTs=1680062133888 2023-03-29 03:55:37,260 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting e0056ddd6eed4fa9981d4d5e16a224aa, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=362, earliestPutTs=1680062136055 2023-03-29 03:55:37,261 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 76d8e08be1494c909ee3994baf678cd9, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=372, earliestPutTs=1680062136702 2023-03-29 03:55:37,267 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] throttle.PressureAwareThroughputController(145): a5c78075cc0f4fd1864c2f0339ad3d67#C#compaction#412 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:55:37,279 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/C/d081df086d5c4cce9a646c3c34c3e866 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/d081df086d5c4cce9a646c3c34c3e866 2023-03-29 03:55:37,283 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in a5c78075cc0f4fd1864c2f0339ad3d67/C of a5c78075cc0f4fd1864c2f0339ad3d67 into d081df086d5c4cce9a646c3c34c3e866(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:55:37,283 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a5c78075cc0f4fd1864c2f0339ad3d67: 2023-03-29 03:55:37,283 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67., storeName=a5c78075cc0f4fd1864c2f0339ad3d67/C, priority=12, startTime=1680062137203; duration=0sec 2023-03-29 03:55:37,283 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:55:37,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on a5c78075cc0f4fd1864c2f0339ad3d67 2023-03-29 03:55:37,474 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. as already flushing 2023-03-29 03:55:37,493 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:37,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51990 deadline: 1680062197487, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:37,500 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:37,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51970 deadline: 1680062197495, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:37,596 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:37,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51990 deadline: 1680062197594, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:37,603 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:37,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51970 deadline: 1680062197601, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:37,668 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303292567b7aa0f2144e9b16fc0ec4c384f65 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303292567b7aa0f2144e9b16fc0ec4c384f65 2023-03-29 03:55:37,675 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=408, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/A/0b0643c298914593801a416407433a62 2023-03-29 03:55:37,685 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=408 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/B/576af5877f6848d6972402acbc61cfe2 2023-03-29 03:55:37,696 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=408 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/C/674edefba30a40b8a4c8013af97c971c 2023-03-29 03:55:37,700 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/A/0b0643c298914593801a416407433a62 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/0b0643c298914593801a416407433a62 2023-03-29 03:55:37,703 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/0b0643c298914593801a416407433a62, entries=150, sequenceid=408, filesize=25.4 K 2023-03-29 03:55:37,704 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/B/576af5877f6848d6972402acbc61cfe2 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/576af5877f6848d6972402acbc61cfe2 2023-03-29 03:55:37,707 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/576af5877f6848d6972402acbc61cfe2, entries=150, sequenceid=408, filesize=11.9 K 2023-03-29 03:55:37,708 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/C/674edefba30a40b8a4c8013af97c971c as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/674edefba30a40b8a4c8013af97c971c 2023-03-29 03:55:37,710 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/674edefba30a40b8a4c8013af97c971c, entries=150, sequenceid=408, filesize=11.9 K 2023-03-29 03:55:37,711 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for a5c78075cc0f4fd1864c2f0339ad3d67 in 467ms, sequenceid=408, compaction requested=false 2023-03-29 03:55:37,711 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a5c78075cc0f4fd1864c2f0339ad3d67: 2023-03-29 03:55:37,711 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:37,711 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:37,711 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:37,711 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:37,711 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:37,711 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:37,713 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:37,713 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:37,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:37,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:37,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:37,713 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:37,713 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:37,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:37,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:37,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:37,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:37,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:37,714 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:37,714 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@40ff8151[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:37,714 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:37,715 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:37,717 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:37,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:37,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:37,717 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:37,717 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:37,717 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:37,717 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:37,717 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:37,719 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:37,719 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:37,719 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:37,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:37,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:37,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:37,719 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:37,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:37,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:37,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:37,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:37,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:37,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:37,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:37,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:37,721 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:37,721 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:37,721 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:37,721 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:37,721 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:37,721 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:37,725 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:37,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:37,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:37,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:37,726 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:37,726 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:37,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:37,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:37,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:37,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:37,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:37,726 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:37,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:37,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:37,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:37,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:37,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:37,727 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:37,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:37,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:37,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:37,730 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:37,730 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:37,730 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:37,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:37,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:37,730 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:37,730 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:37,730 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:37,731 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:37,731 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:37,731 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:37,731 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:37,730 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:37,730 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:37,731 DEBUG [Thread-2498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:37,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:37,731 DEBUG [Thread-2498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:37,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:37,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on a5c78075cc0f4fd1864c2f0339ad3d67 2023-03-29 03:55:37,802 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a5c78075cc0f4fd1864c2f0339ad3d67 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-29 03:55:37,802 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5c78075cc0f4fd1864c2f0339ad3d67, store=A 2023-03-29 03:55:37,802 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:55:37,802 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5c78075cc0f4fd1864c2f0339ad3d67, store=B 2023-03-29 03:55:37,802 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:55:37,802 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5c78075cc0f4fd1864c2f0339ad3d67, store=C 2023-03-29 03:55:37,802 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:55:37,831 DEBUG [Thread-2498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:37,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:37,833 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:37,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:37,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:37,834 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:37,834 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:37,834 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:37,834 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:37,835 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:37,835 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:37,836 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:37,836 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:37,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:37,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:37,836 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:37,836 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:37,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:37,837 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:37,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:37,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:37,837 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:37,837 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:37,837 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:37,837 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:37,837 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:37,837 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:37,837 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:37,837 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:37,838 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:37,838 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. started... 2023-03-29 03:55:37,838 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. as already flushing 2023-03-29 03:55:37,838 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:37,838 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:37,838 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:37,838 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:37,838 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:37,838 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:37,839 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:37,839 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:37,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:37,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:37,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:37,840 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:37,840 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:37,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:37,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:37,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:37,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:37,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:37,842 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:37,842 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5298b59f[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:37,842 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:37,842 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:37,843 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:37,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:37,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:37,843 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:37,843 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:37,843 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:37,843 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:37,843 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:37,846 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:37,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:37,846 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:37,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:37,846 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:37,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:37,846 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:37,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:37,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:37,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:37,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:37,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:37,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:37,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:37,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:37,848 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:37,848 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:37,848 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:37,848 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:37,848 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:37,848 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:37,849 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:37,849 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:37,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:37,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:37,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:37,849 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:37,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:37,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:37,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:37,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:37,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:37,850 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:37,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:37,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:37,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:37,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:37,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:37,853 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:37,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:37,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:37,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:37,855 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:37,855 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:37,855 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:37,855 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:55:37,855 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:37,856 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:37,855 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:37,856 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:37,856 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:37,856 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:37,856 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:37,856 DEBUG [Thread-2498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:37,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:37,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:37,855 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:37,856 DEBUG [Thread-2498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:37,856 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:37,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:37,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:37,866 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680062137840, max timestamp=1680062137864Num uniques 859; 2023-03-29 03:55:37,866 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-29 03:55:37,866 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680062137840, max timestamp=1680062137864Num uniques 859; 2023-03-29 03:55:37,866 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-29 03:55:37,866 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680062137840, max timestamp=1680062137864Num uniques 859; 2023-03-29 03:55:37,866 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-29 03:55:37,921 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:37,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51990 deadline: 1680062197912, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:37,921 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:37,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51970 deadline: 1680062197912, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:37,956 DEBUG [Thread-2498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:37,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:37,958 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:37,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:37,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:37,959 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:37,959 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:37,959 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:37,959 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:37,959 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:37,959 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:37,961 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:37,961 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:37,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:37,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:37,961 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:37,961 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:37,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:37,961 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:37,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:37,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:37,962 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:37,962 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:37,962 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:37,962 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:37,962 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:37,962 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:37,962 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:37,962 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:37,962 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:37,962 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. started... 2023-03-29 03:55:37,963 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. as already flushing 2023-03-29 03:55:37,963 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:37,963 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:37,963 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:37,963 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:37,963 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:37,963 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:37,965 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:37,965 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:37,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:37,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:37,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:37,965 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:37,965 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:37,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:37,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:37,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:37,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:37,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:37,966 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:37,966 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@73baa17a[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:37,966 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:37,966 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:37,968 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:37,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:37,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:37,968 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:37,968 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:37,968 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:37,968 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:37,968 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:37,969 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:37,969 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:37,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:37,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:37,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:37,969 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:37,969 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:37,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:37,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:37,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:37,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:37,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:37,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:37,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:37,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:37,981 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:37,981 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:37,981 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:37,981 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:37,981 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:37,982 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:37,983 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:37,983 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:37,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:37,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:37,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:37,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:37,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:37,983 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:37,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:37,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:37,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:37,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:37,984 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:37,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:37,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:37,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:37,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:37,985 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:37,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:37,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:37,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:37,990 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:37,990 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:37,990 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:37,990 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:37,990 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:37,990 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:37,990 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:37,990 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:37,990 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:37,990 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:37,990 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:37,990 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:37,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:37,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:37,990 DEBUG [Thread-2498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:37,990 DEBUG [Thread-2498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:37,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:37,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:38,026 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:38,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51990 deadline: 1680062198022, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:38,026 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:38,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51970 deadline: 1680062198022, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:38,078 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/A/26b04e139e5e45fb98bff564bafc4c46 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/26b04e139e5e45fb98bff564bafc4c46 2023-03-29 03:55:38,082 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in a5c78075cc0f4fd1864c2f0339ad3d67/A of a5c78075cc0f4fd1864c2f0339ad3d67 into 26b04e139e5e45fb98bff564bafc4c46(size=26.2 K), total size for store is 51.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:55:38,082 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a5c78075cc0f4fd1864c2f0339ad3d67: 2023-03-29 03:55:38,082 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67., storeName=a5c78075cc0f4fd1864c2f0339ad3d67/A, priority=12, startTime=1680062137202; duration=0sec 2023-03-29 03:55:38,082 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:55:38,090 DEBUG [Thread-2498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:38,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:38,093 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:38,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:38,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:38,094 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:38,094 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:38,094 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:38,094 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:38,095 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:38,095 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:38,096 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:38,096 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:38,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:38,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:38,096 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:38,097 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:38,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:38,097 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:38,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:38,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:38,097 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:38,098 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:38,099 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:38,099 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:38,099 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:38,099 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:38,099 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:38,099 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:38,099 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. started... 2023-03-29 03:55:38,099 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. as already flushing 2023-03-29 03:55:38,099 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:38,099 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:38,099 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:38,099 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:38,099 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:38,099 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:38,101 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:38,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:38,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:38,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:38,101 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:38,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:38,102 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:38,102 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:38,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:38,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:38,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:38,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:38,104 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:38,104 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b9eb2cf[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:38,104 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:38,104 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:38,105 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:38,106 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:38,106 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:38,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:38,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:38,106 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:38,106 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:38,106 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:38,109 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:38,109 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:38,109 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:38,109 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:38,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:38,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:38,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:38,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:38,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:38,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:38,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:38,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:38,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:38,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:38,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:38,114 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:38,114 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:38,114 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:38,114 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:38,114 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:38,114 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:38,115 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:38,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:38,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:38,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:38,116 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:38,116 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:38,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:38,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:38,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:38,116 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:38,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:38,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:38,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:38,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:38,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:38,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:38,117 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:38,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:38,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:38,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:38,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:38,124 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:38,124 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:38,124 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:38,124 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:38,124 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:38,124 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:38,124 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:38,124 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:38,124 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:38,124 DEBUG [Thread-2498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:38,124 DEBUG [Thread-2498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:38,124 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:38,124 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:55:38,124 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:38,125 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:38,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:38,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:38,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:38,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:38,214 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230329433df01ad1984d13a4a865d8ebc4a19e to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230329433df01ad1984d13a4a865d8ebc4a19e 2023-03-29 03:55:38,219 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=421, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/A/d4e0b1bad0264614b0db6496f8d4732c 2023-03-29 03:55:38,224 DEBUG [Thread-2498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:38,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:38,227 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:38,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:38,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:38,228 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:38,228 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:38,228 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:38,228 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:38,228 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:38,229 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:38,229 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=421 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/B/277e74dc0d214ca5a5d3cfc1d2588ce3 2023-03-29 03:55:38,232 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:38,232 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:38,232 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:38,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:38,232 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:38,232 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:38,233 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:38,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 246 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51990 deadline: 1680062198227, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:38,233 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:38,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:38,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51970 deadline: 1680062198228, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:38,233 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:38,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:38,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:38,234 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:38,234 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:38,234 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:38,235 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:38,235 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:38,235 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:38,235 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:38,235 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:38,235 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:38,235 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. started... 2023-03-29 03:55:38,235 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. as already flushing 2023-03-29 03:55:38,235 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:38,236 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:38,236 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:38,236 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:38,236 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:38,236 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:38,238 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:38,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:38,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:38,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:38,238 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:38,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:38,239 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:38,239 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:38,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:38,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:38,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:38,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:38,240 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:38,240 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@775757e4[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:38,240 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:38,240 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:38,241 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:38,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:38,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:38,241 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:38,241 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:38,241 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:38,241 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:38,241 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:38,246 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:38,246 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:38,246 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:38,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:38,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:38,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:38,246 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:38,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:38,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:38,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:38,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:38,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:38,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:38,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:38,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:38,248 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:38,248 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:38,248 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:38,248 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:38,248 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:38,248 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:38,250 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:38,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:38,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:38,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:38,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:38,250 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:38,251 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:38,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:38,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:38,252 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:38,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:38,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:38,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:38,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:38,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:38,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:38,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:38,253 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:38,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:38,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:38,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:38,255 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:38,255 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:38,255 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:38,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:38,255 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:55:38,255 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:38,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:38,255 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:38,255 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:38,255 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:38,255 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:38,256 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:38,256 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:38,256 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:38,256 DEBUG [Thread-2498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:38,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:38,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:38,256 DEBUG [Thread-2498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:38,256 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:38,356 DEBUG [Thread-2498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:38,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:38,358 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:38,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:38,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:38,359 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:38,359 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:38,359 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:38,359 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:38,359 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:38,359 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:38,361 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:38,361 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:38,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:38,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:38,361 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:38,361 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:38,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:38,361 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:38,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:38,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:38,362 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:38,362 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:38,362 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:38,362 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:38,362 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:38,362 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:38,362 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:38,362 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:38,362 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:38,362 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. started... 2023-03-29 03:55:38,362 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. as already flushing 2023-03-29 03:55:38,362 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:38,362 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:38,362 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:38,363 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:38,363 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:38,363 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:38,365 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:38,365 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:38,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:38,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:38,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:38,365 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:38,365 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:38,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:38,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:38,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:38,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:38,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:38,366 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:38,366 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@607430dc[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:38,366 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:38,366 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:38,368 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:38,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:38,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:38,368 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:38,368 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:38,368 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:38,368 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:38,368 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:38,369 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:38,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:38,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:38,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:38,369 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:38,369 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:38,369 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:38,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:38,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:38,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:38,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:38,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:38,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:38,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:38,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:38,371 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:38,371 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:38,371 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:38,372 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:38,372 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:38,372 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:38,373 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:38,373 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:38,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:38,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:38,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:38,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:38,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:38,373 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:38,374 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:38,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:38,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:38,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:38,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:38,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:38,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:38,375 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:38,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:38,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:38,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:38,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:38,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:38,383 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:38,383 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:38,383 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:38,383 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:38,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:38,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:38,383 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:38,383 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:38,383 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:38,383 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:55:38,383 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:38,383 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:38,383 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:38,383 DEBUG [Thread-2498] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:38,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:38,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:38,383 DEBUG [Thread-2498] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:38,383 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:38,384 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:38,470 DEBUG [Thread-2503] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1ec12b02 to 127.0.0.1:52103 2023-03-29 03:55:38,470 DEBUG [Thread-2503] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-29 03:55:38,476 DEBUG [Thread-2505] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1d7ba242 to 127.0.0.1:52103 2023-03-29 03:55:38,476 DEBUG [Thread-2505] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-29 03:55:38,476 DEBUG [Thread-2507] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3e079da1 to 127.0.0.1:52103 2023-03-29 03:55:38,476 DEBUG [Thread-2507] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-29 03:55:38,477 DEBUG [Thread-2501] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x68a0e21a to 127.0.0.1:52103 2023-03-29 03:55:38,477 DEBUG [Thread-2501] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-29 03:55:38,483 DEBUG [Thread-2499] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x51ccd8b7 to 127.0.0.1:52103 2023-03-29 03:55:38,483 DEBUG [Thread-2499] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-29 03:55:38,484 DEBUG [Thread-2498] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:38,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:38,534 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:38,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 246 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51970 deadline: 1680062198534, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:38,536 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:38,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 248 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:51990 deadline: 1680062198536, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:38,639 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=421 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/C/7ccbcd2821f44f059c0458fa44806dbc 2023-03-29 03:55:38,642 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/A/d4e0b1bad0264614b0db6496f8d4732c as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/d4e0b1bad0264614b0db6496f8d4732c 2023-03-29 03:55:38,645 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/d4e0b1bad0264614b0db6496f8d4732c, entries=200, sequenceid=421, filesize=32.3 K 2023-03-29 03:55:38,645 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/B/277e74dc0d214ca5a5d3cfc1d2588ce3 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/277e74dc0d214ca5a5d3cfc1d2588ce3 2023-03-29 03:55:38,648 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/277e74dc0d214ca5a5d3cfc1d2588ce3, entries=150, sequenceid=421, filesize=11.9 K 2023-03-29 03:55:38,649 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/C/7ccbcd2821f44f059c0458fa44806dbc as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/7ccbcd2821f44f059c0458fa44806dbc 2023-03-29 03:55:38,651 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/7ccbcd2821f44f059c0458fa44806dbc, entries=150, sequenceid=421, filesize=11.9 K 2023-03-29 03:55:38,652 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=136.42 KB/139690 for a5c78075cc0f4fd1864c2f0339ad3d67 in 850ms, sequenceid=421, compaction requested=true 2023-03-29 03:55:38,652 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a5c78075cc0f4fd1864c2f0339ad3d67: 2023-03-29 03:55:38,652 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:55:38,652 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:55:38,652 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-29 03:55:38,652 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-29 03:55:38,652 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:55:38,653 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 85962 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-29 03:55:38,653 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37342 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-29 03:55:38,653 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1912): a5c78075cc0f4fd1864c2f0339ad3d67/A is initiating minor compaction (all files) 2023-03-29 03:55:38,653 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1912): a5c78075cc0f4fd1864c2f0339ad3d67/B is initiating minor compaction (all files) 2023-03-29 03:55:38,653 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a5c78075cc0f4fd1864c2f0339ad3d67/A in TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:38,653 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a5c78075cc0f4fd1864c2f0339ad3d67/B in TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:38,653 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/26b04e139e5e45fb98bff564bafc4c46, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/0b0643c298914593801a416407433a62, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/d4e0b1bad0264614b0db6496f8d4732c] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp, totalSize=83.9 K 2023-03-29 03:55:38,653 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/d308d6c26d3d436cbd990b94a914c6c7, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/576af5877f6848d6972402acbc61cfe2, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/277e74dc0d214ca5a5d3cfc1d2588ce3] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp, totalSize=36.5 K 2023-03-29 03:55:38,653 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 26b04e139e5e45fb98bff564bafc4c46, keycount=150, bloomtype=ROW, size=26.2 K, encoding=NONE, compression=NONE, seqNum=372, earliestPutTs=1680062136063 2023-03-29 03:55:38,654 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting d308d6c26d3d436cbd990b94a914c6c7, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=372, earliestPutTs=1680062136063 2023-03-29 03:55:38,654 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 0b0643c298914593801a416407433a62, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=408, earliestPutTs=1680062136848 2023-03-29 03:55:38,654 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 576af5877f6848d6972402acbc61cfe2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=408, earliestPutTs=1680062136848 2023-03-29 03:55:38,654 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting d4e0b1bad0264614b0db6496f8d4732c, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=421, earliestPutTs=1680062137485 2023-03-29 03:55:38,654 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 277e74dc0d214ca5a5d3cfc1d2588ce3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=421, earliestPutTs=1680062137485 2023-03-29 03:55:38,659 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] throttle.PressureAwareThroughputController(145): a5c78075cc0f4fd1864c2f0339ad3d67#B#compaction#419 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:55:38,661 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] throttle.PressureAwareThroughputController(145): a5c78075cc0f4fd1864c2f0339ad3d67#A#compaction#418 average throughput is 6.57 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:55:39,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on a5c78075cc0f4fd1864c2f0339ad3d67 2023-03-29 03:55:39,039 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a5c78075cc0f4fd1864c2f0339ad3d67 3/3 column families, dataSize=149.83 KB heapSize=430.61 KB 2023-03-29 03:55:39,039 DEBUG [Thread-2492] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x71b5058b to 127.0.0.1:52103 2023-03-29 03:55:39,040 DEBUG [Thread-2490] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x38341134 to 127.0.0.1:52103 2023-03-29 03:55:39,040 DEBUG [Thread-2492] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-29 03:55:39,040 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5c78075cc0f4fd1864c2f0339ad3d67, store=A 2023-03-29 03:55:39,040 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-29 03:55:39,040 DEBUG [Thread-2490] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-29 03:55:39,040 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5c78075cc0f4fd1864c2f0339ad3d67, store=B 2023-03-29 03:55:39,040 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-29 03:55:39,040 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5c78075cc0f4fd1864c2f0339ad3d67, store=C 2023-03-29 03:55:39,040 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-29 03:55:39,066 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/B/6ca81f3168ce4f368d5963cc7ba52510 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/6ca81f3168ce4f368d5963cc7ba52510 2023-03-29 03:55:39,069 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a5c78075cc0f4fd1864c2f0339ad3d67/B of a5c78075cc0f4fd1864c2f0339ad3d67 into 6ca81f3168ce4f368d5963cc7ba52510(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:55:39,069 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a5c78075cc0f4fd1864c2f0339ad3d67: 2023-03-29 03:55:39,069 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67., storeName=a5c78075cc0f4fd1864c2f0339ad3d67/B, priority=13, startTime=1680062138652; duration=0sec 2023-03-29 03:55:39,069 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:55:39,070 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-29 03:55:39,071 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37342 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-29 03:55:39,071 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1912): a5c78075cc0f4fd1864c2f0339ad3d67/C is initiating minor compaction (all files) 2023-03-29 03:55:39,071 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a5c78075cc0f4fd1864c2f0339ad3d67/C in TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:39,071 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/d081df086d5c4cce9a646c3c34c3e866, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/674edefba30a40b8a4c8013af97c971c, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/7ccbcd2821f44f059c0458fa44806dbc] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp, totalSize=36.5 K 2023-03-29 03:55:39,071 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting d081df086d5c4cce9a646c3c34c3e866, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=372, earliestPutTs=1680062136063 2023-03-29 03:55:39,071 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 674edefba30a40b8a4c8013af97c971c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=408, earliestPutTs=1680062136848 2023-03-29 03:55:39,072 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 7ccbcd2821f44f059c0458fa44806dbc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=421, earliestPutTs=1680062137485 2023-03-29 03:55:39,077 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/A/bbf8e1beffc540a89c62d402ed7ac930 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/bbf8e1beffc540a89c62d402ed7ac930 2023-03-29 03:55:39,078 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] throttle.PressureAwareThroughputController(145): a5c78075cc0f4fd1864c2f0339ad3d67#C#compaction#421 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:55:39,080 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a5c78075cc0f4fd1864c2f0339ad3d67/A of a5c78075cc0f4fd1864c2f0339ad3d67 into bbf8e1beffc540a89c62d402ed7ac930(size=26.3 K), total size for store is 26.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:55:39,080 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a5c78075cc0f4fd1864c2f0339ad3d67: 2023-03-29 03:55:39,080 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67., storeName=a5c78075cc0f4fd1864c2f0339ad3d67/A, priority=13, startTime=1680062138652; duration=0sec 2023-03-29 03:55:39,080 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:55:39,084 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/C/527f0f58cd2840e7b15292e60d5c695a as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/527f0f58cd2840e7b15292e60d5c695a 2023-03-29 03:55:39,087 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a5c78075cc0f4fd1864c2f0339ad3d67/C of a5c78075cc0f4fd1864c2f0339ad3d67 into 527f0f58cd2840e7b15292e60d5c695a(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:55:39,087 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a5c78075cc0f4fd1864c2f0339ad3d67: 2023-03-29 03:55:39,087 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67., storeName=a5c78075cc0f4fd1864c2f0339ad3d67/C, priority=13, startTime=1680062138652; duration=0sec 2023-03-29 03:55:39,087 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:55:39,451 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032963af062249ca48cbb8704fd4236e3376 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032963af062249ca48cbb8704fd4236e3376 2023-03-29 03:55:39,622 DEBUG [Thread-2496] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4fe1db6a to 127.0.0.1:52103 2023-03-29 03:55:39,622 DEBUG [Thread-2496] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-29 03:55:39,669 DEBUG [Thread-2494] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3620ee35 to 127.0.0.1:52103 2023-03-29 03:55:39,669 DEBUG [Thread-2494] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-29 03:55:39,708 DEBUG [Thread-2488] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0bcfc1e0 to 127.0.0.1:52103 2023-03-29 03:55:39,708 DEBUG [Thread-2488] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-29 03:55:39,708 INFO [Listener at localhost/35027] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-03-29 03:55:39,708 INFO [Listener at localhost/35027] hbase.AcidGuaranteesTestTool(394): wrote 25 2023-03-29 03:55:39,708 INFO [Listener at localhost/35027] hbase.AcidGuaranteesTestTool(394): wrote 147 2023-03-29 03:55:39,708 INFO [Listener at localhost/35027] hbase.AcidGuaranteesTestTool(394): wrote 149 2023-03-29 03:55:39,708 INFO [Listener at localhost/35027] hbase.AcidGuaranteesTestTool(394): wrote 24 2023-03-29 03:55:39,708 INFO [Listener at localhost/35027] hbase.AcidGuaranteesTestTool(394): wrote 26 2023-03-29 03:55:39,708 INFO [Listener at localhost/35027] hbase.AcidGuaranteesTestTool(396): Readers: 2023-03-29 03:55:39,708 INFO [Listener at localhost/35027] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-03-29 03:55:39,708 INFO [Listener at localhost/35027] hbase.AcidGuaranteesTestTool(402): scanned 1808 2023-03-29 03:55:39,708 INFO [Listener at localhost/35027] hbase.AcidGuaranteesTestTool(403): verified 5424 rows 2023-03-29 03:55:39,708 INFO [Listener at localhost/35027] hbase.AcidGuaranteesTestTool(402): scanned 1817 2023-03-29 03:55:39,708 INFO [Listener at localhost/35027] hbase.AcidGuaranteesTestTool(403): verified 5444 rows 2023-03-29 03:55:39,708 INFO [Listener at localhost/35027] hbase.AcidGuaranteesTestTool(402): scanned 1805 2023-03-29 03:55:39,708 INFO [Listener at localhost/35027] hbase.AcidGuaranteesTestTool(403): verified 5414 rows 2023-03-29 03:55:39,709 INFO [Listener at localhost/35027] hbase.AcidGuaranteesTestTool(402): scanned 1805 2023-03-29 03:55:39,709 INFO [Listener at localhost/35027] hbase.AcidGuaranteesTestTool(403): verified 5414 rows 2023-03-29 03:55:39,709 INFO [Listener at localhost/35027] hbase.AcidGuaranteesTestTool(402): scanned 1810 2023-03-29 03:55:39,709 INFO [Listener at localhost/35027] hbase.AcidGuaranteesTestTool(403): verified 5429 rows 2023-03-29 03:55:39,709 INFO [Listener at localhost/35027] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-03-29 03:55:39,709 DEBUG [Listener at localhost/35027] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1a5ab074 to 127.0.0.1:52103 2023-03-29 03:55:39,709 DEBUG [Listener at localhost/35027] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-29 03:55:39,712 INFO [Listener at localhost/35027] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-03-29 03:55:39,712 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-03-29 03:55:39,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure2.ProcedureExecutor(1029): Stored pid=43, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-03-29 03:55:39,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-03-29 03:55:39,716 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680062139716"}]},"ts":"1680062139716"} 2023-03-29 03:55:39,717 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-03-29 03:55:39,813 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-03-29 03:55:39,814 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=44, ppid=43, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=a5c78075cc0f4fd1864c2f0339ad3d67, UNASSIGN}] 2023-03-29 03:55:39,815 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, ppid=43, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=a5c78075cc0f4fd1864c2f0339ad3d67, UNASSIGN 2023-03-29 03:55:39,816 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=44 updating hbase:meta row=a5c78075cc0f4fd1864c2f0339ad3d67, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:39,816 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680062139816"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680062139816"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680062139816"}]},"ts":"1680062139816"} 2023-03-29 03:55:39,817 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=45, ppid=44, state=RUNNABLE; CloseRegionProcedure a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534}] 2023-03-29 03:55:39,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-03-29 03:55:39,855 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=461, memsize=51.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/A/05a5587259004dadb5b6943828037619 2023-03-29 03:55:39,887 DEBUG [regionserver/jenkins-hbase4:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.0, tune throughput to 50.00 MB/second 2023-03-29 03:55:39,906 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/413b333ae3934688bf785f28dd43be25, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/99006d66cc3e432b8e2abc2436babbc6, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/cafee5e0813f4a2da3cd17f07119e801, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/6ee45ee7b9204e39933bc2b40c5a9b19, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/b815d2bcbd3a487ba420d75dddbe1c6d, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/dd266c63ee75421183eefedfab4f43c9, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/262d9d63ebaa47208cfe6461c39ece00, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/5cccd1ba78084bb8bbb93b502fa577bd, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/217fd2cf2cf44201b12572d4b9218c9d, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/496176f4abe44d389b39e070052d006f, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/8848dbbad301430ea69d561a16c84f9f, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/ad7ef9e3d87d40f7accb56d92b20437d, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/d736a34884574dada571d2b9d7c1b07c, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/cf6aa67f579649e7b0fe9f2277172a14, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/5f65fa1579ec4cc6894a148c8d3fa060, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/8cf0f8e20dd94b87b97107cb27a9a2e9, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/56dfba20777f471b83b8bb998260a06c, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/3c797965921e42cca36338c3c3f382af, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/6ab685bea079460fba8340e942e7f966, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/94556f512cbc482dafab99e54df84c91, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/5a9a313901454e239923a9dc8e82b169, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/c042bfc59f5c4a119d7c895b8feef858, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/514dbc28359f4e2abc2a1f9dcd0a3732, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/26b04e139e5e45fb98bff564bafc4c46, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/0b0643c298914593801a416407433a62, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/d4e0b1bad0264614b0db6496f8d4732c] to archive 2023-03-29 03:55:39,906 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-03-29 03:55:39,908 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/413b333ae3934688bf785f28dd43be25 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/413b333ae3934688bf785f28dd43be25 2023-03-29 03:55:39,909 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/99006d66cc3e432b8e2abc2436babbc6 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/99006d66cc3e432b8e2abc2436babbc6 2023-03-29 03:55:39,910 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/cafee5e0813f4a2da3cd17f07119e801 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/cafee5e0813f4a2da3cd17f07119e801 2023-03-29 03:55:39,910 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/6ee45ee7b9204e39933bc2b40c5a9b19 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/6ee45ee7b9204e39933bc2b40c5a9b19 2023-03-29 03:55:39,911 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/b815d2bcbd3a487ba420d75dddbe1c6d to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/b815d2bcbd3a487ba420d75dddbe1c6d 2023-03-29 03:55:39,912 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/dd266c63ee75421183eefedfab4f43c9 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/dd266c63ee75421183eefedfab4f43c9 2023-03-29 03:55:39,913 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/262d9d63ebaa47208cfe6461c39ece00 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/262d9d63ebaa47208cfe6461c39ece00 2023-03-29 03:55:39,914 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/5cccd1ba78084bb8bbb93b502fa577bd to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/5cccd1ba78084bb8bbb93b502fa577bd 2023-03-29 03:55:39,915 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/217fd2cf2cf44201b12572d4b9218c9d to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/217fd2cf2cf44201b12572d4b9218c9d 2023-03-29 03:55:39,915 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/496176f4abe44d389b39e070052d006f to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/496176f4abe44d389b39e070052d006f 2023-03-29 03:55:39,916 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/8848dbbad301430ea69d561a16c84f9f to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/8848dbbad301430ea69d561a16c84f9f 2023-03-29 03:55:39,917 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/ad7ef9e3d87d40f7accb56d92b20437d to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/ad7ef9e3d87d40f7accb56d92b20437d 2023-03-29 03:55:39,918 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/d736a34884574dada571d2b9d7c1b07c to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/d736a34884574dada571d2b9d7c1b07c 2023-03-29 03:55:39,918 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/cf6aa67f579649e7b0fe9f2277172a14 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/cf6aa67f579649e7b0fe9f2277172a14 2023-03-29 03:55:39,919 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/5f65fa1579ec4cc6894a148c8d3fa060 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/5f65fa1579ec4cc6894a148c8d3fa060 2023-03-29 03:55:39,920 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/8cf0f8e20dd94b87b97107cb27a9a2e9 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/8cf0f8e20dd94b87b97107cb27a9a2e9 2023-03-29 03:55:39,921 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/56dfba20777f471b83b8bb998260a06c to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/56dfba20777f471b83b8bb998260a06c 2023-03-29 03:55:39,922 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/3c797965921e42cca36338c3c3f382af to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/3c797965921e42cca36338c3c3f382af 2023-03-29 03:55:39,923 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/6ab685bea079460fba8340e942e7f966 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/6ab685bea079460fba8340e942e7f966 2023-03-29 03:55:39,924 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/94556f512cbc482dafab99e54df84c91 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/94556f512cbc482dafab99e54df84c91 2023-03-29 03:55:39,925 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/5a9a313901454e239923a9dc8e82b169 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/5a9a313901454e239923a9dc8e82b169 2023-03-29 03:55:39,926 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/c042bfc59f5c4a119d7c895b8feef858 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/c042bfc59f5c4a119d7c895b8feef858 2023-03-29 03:55:39,926 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/514dbc28359f4e2abc2a1f9dcd0a3732 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/514dbc28359f4e2abc2a1f9dcd0a3732 2023-03-29 03:55:39,927 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/26b04e139e5e45fb98bff564bafc4c46 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/26b04e139e5e45fb98bff564bafc4c46 2023-03-29 03:55:39,928 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/0b0643c298914593801a416407433a62 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/0b0643c298914593801a416407433a62 2023-03-29 03:55:39,929 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/d4e0b1bad0264614b0db6496f8d4732c to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/d4e0b1bad0264614b0db6496f8d4732c 2023-03-29 03:55:39,942 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/c79a2874dd184bcba8eb67a074c63a88, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/15a8b139d38f457d9284e7e2e260d532, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/5052efffd2a84a8d9080bcba50d3a88b, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/13fdff8451574686ab05e509355583bf, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/d45741edc43e4f80a0e4ad2e592dfc6a, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/f18327be7b684c08af91125d8c768797, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/2314c1fead8842ccb2a38abfbd527a3d, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/7dff4f1e20944e93a93900eaae922eff, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/449af9b48e734a02b5e692557b1a1e17, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/d1a227f3118c40498b880f98c929c7b7, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/a2e31e392f844f6698bae1872e0f9b15, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/4ef04e1fe5da4761afd2ece4d4fc9d1e, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/be95af63faee4fdbb7ba85abfa3ba828, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/5860d676a0af4102a1ab89c009bafdfb, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/c5866dee61e44f8f914008097aa0e9e1, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/e266a7c7073742c3a0f270bb8ed0bc41, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/d239385552404f1c8762ef7d80ec858c, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/2aa68d67535d414bb5f9875f7bf90a93, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/7e94db7fe04348f6b81bc03a9e88ff21, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/7ec4b24c6ebd41f2b2703f5cc530b77e, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/be951641a9b144dfb7cc65cf3d8c8555, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/d28acf570c834d2e83f4de9473c9a6c5, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/d308d6c26d3d436cbd990b94a914c6c7, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/a1646ba23916429eb570a4f4da223d52, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/576af5877f6848d6972402acbc61cfe2, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/277e74dc0d214ca5a5d3cfc1d2588ce3] to archive 2023-03-29 03:55:39,943 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-03-29 03:55:39,944 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/c79a2874dd184bcba8eb67a074c63a88 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/c79a2874dd184bcba8eb67a074c63a88 2023-03-29 03:55:39,945 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/15a8b139d38f457d9284e7e2e260d532 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/15a8b139d38f457d9284e7e2e260d532 2023-03-29 03:55:39,946 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/5052efffd2a84a8d9080bcba50d3a88b to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/5052efffd2a84a8d9080bcba50d3a88b 2023-03-29 03:55:39,947 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/13fdff8451574686ab05e509355583bf to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/13fdff8451574686ab05e509355583bf 2023-03-29 03:55:39,947 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/d45741edc43e4f80a0e4ad2e592dfc6a to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/d45741edc43e4f80a0e4ad2e592dfc6a 2023-03-29 03:55:39,948 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/f18327be7b684c08af91125d8c768797 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/f18327be7b684c08af91125d8c768797 2023-03-29 03:55:39,949 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/2314c1fead8842ccb2a38abfbd527a3d to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/2314c1fead8842ccb2a38abfbd527a3d 2023-03-29 03:55:39,950 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/7dff4f1e20944e93a93900eaae922eff to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/7dff4f1e20944e93a93900eaae922eff 2023-03-29 03:55:39,950 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/449af9b48e734a02b5e692557b1a1e17 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/449af9b48e734a02b5e692557b1a1e17 2023-03-29 03:55:39,951 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/d1a227f3118c40498b880f98c929c7b7 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/d1a227f3118c40498b880f98c929c7b7 2023-03-29 03:55:39,952 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/a2e31e392f844f6698bae1872e0f9b15 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/a2e31e392f844f6698bae1872e0f9b15 2023-03-29 03:55:39,952 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/4ef04e1fe5da4761afd2ece4d4fc9d1e to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/4ef04e1fe5da4761afd2ece4d4fc9d1e 2023-03-29 03:55:39,953 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/be95af63faee4fdbb7ba85abfa3ba828 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/be95af63faee4fdbb7ba85abfa3ba828 2023-03-29 03:55:39,954 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/5860d676a0af4102a1ab89c009bafdfb to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/5860d676a0af4102a1ab89c009bafdfb 2023-03-29 03:55:39,955 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/c5866dee61e44f8f914008097aa0e9e1 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/c5866dee61e44f8f914008097aa0e9e1 2023-03-29 03:55:39,955 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/e266a7c7073742c3a0f270bb8ed0bc41 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/e266a7c7073742c3a0f270bb8ed0bc41 2023-03-29 03:55:39,956 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/d239385552404f1c8762ef7d80ec858c to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/d239385552404f1c8762ef7d80ec858c 2023-03-29 03:55:39,957 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/2aa68d67535d414bb5f9875f7bf90a93 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/2aa68d67535d414bb5f9875f7bf90a93 2023-03-29 03:55:39,958 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/7e94db7fe04348f6b81bc03a9e88ff21 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/7e94db7fe04348f6b81bc03a9e88ff21 2023-03-29 03:55:39,959 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/7ec4b24c6ebd41f2b2703f5cc530b77e to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/7ec4b24c6ebd41f2b2703f5cc530b77e 2023-03-29 03:55:39,960 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/be951641a9b144dfb7cc65cf3d8c8555 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/be951641a9b144dfb7cc65cf3d8c8555 2023-03-29 03:55:39,960 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/d28acf570c834d2e83f4de9473c9a6c5 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/d28acf570c834d2e83f4de9473c9a6c5 2023-03-29 03:55:39,961 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/d308d6c26d3d436cbd990b94a914c6c7 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/d308d6c26d3d436cbd990b94a914c6c7 2023-03-29 03:55:39,962 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/a1646ba23916429eb570a4f4da223d52 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/a1646ba23916429eb570a4f4da223d52 2023-03-29 03:55:39,963 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/576af5877f6848d6972402acbc61cfe2 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/576af5877f6848d6972402acbc61cfe2 2023-03-29 03:55:39,964 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/277e74dc0d214ca5a5d3cfc1d2588ce3 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/277e74dc0d214ca5a5d3cfc1d2588ce3 2023-03-29 03:55:39,968 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close a5c78075cc0f4fd1864c2f0339ad3d67 2023-03-29 03:55:39,968 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing a5c78075cc0f4fd1864c2f0339ad3d67, disabling compactions & flushes 2023-03-29 03:55:39,969 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1858): waiting for 0 compactions & cache flush to complete for region TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:39,969 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/7522e7a58f7749b4a733fc2e1bae07af, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/528af543643c4eed9cf12d5c2aa68c5e, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/dc646a8263e54b96b5315974d304b5af, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/eda0e120d6fa4644820b8f8b43605329, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/ebbc1e200ac34ceba35bf0c8c97c8c8e, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/e25bcd1d2d7b47a7a2a6c9a3172fe613, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/45dba969a2284d319061baff17009a57, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/88b32e3389054c6a9caf2e91ea917865, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/5e7b29bba65940acabd3c8d28c38897d, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/002ddcbbb4c84fe58514457890b34e3d, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/d0ed71e2627849f195cefc1eefb1ff72, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/e89c41ae679d43748bade17b8b1a1144, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/52b982e75b9a42089f729d3115dcfadd, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/15dee25bd8cd419c9c0971ac2d17662f, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/3285c402839d452589d36b939084301f, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/fdd78514d3564102ba146d3c7df2cc02, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/ee69bc58c4634710ac8e3827656bfc6b, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/08b2669d45bd4ab9b031f7107d833fbb, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/2f3277cda34c4a89a04edc2000b805cd, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/72d73d4d55da4fd0a1bddb1f7fbf7dd9, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/a2ceed1c461c495287a0cd19269dd47d, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/e0056ddd6eed4fa9981d4d5e16a224aa, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/d081df086d5c4cce9a646c3c34c3e866, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/76d8e08be1494c909ee3994baf678cd9, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/674edefba30a40b8a4c8013af97c971c, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/7ccbcd2821f44f059c0458fa44806dbc] to archive 2023-03-29 03:55:39,970 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-03-29 03:55:39,971 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/7522e7a58f7749b4a733fc2e1bae07af to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/7522e7a58f7749b4a733fc2e1bae07af 2023-03-29 03:55:39,971 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/528af543643c4eed9cf12d5c2aa68c5e to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/528af543643c4eed9cf12d5c2aa68c5e 2023-03-29 03:55:39,972 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/dc646a8263e54b96b5315974d304b5af to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/dc646a8263e54b96b5315974d304b5af 2023-03-29 03:55:39,973 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/eda0e120d6fa4644820b8f8b43605329 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/eda0e120d6fa4644820b8f8b43605329 2023-03-29 03:55:39,974 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/ebbc1e200ac34ceba35bf0c8c97c8c8e to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/ebbc1e200ac34ceba35bf0c8c97c8c8e 2023-03-29 03:55:39,974 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/e25bcd1d2d7b47a7a2a6c9a3172fe613 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/e25bcd1d2d7b47a7a2a6c9a3172fe613 2023-03-29 03:55:39,975 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/45dba969a2284d319061baff17009a57 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/45dba969a2284d319061baff17009a57 2023-03-29 03:55:39,976 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/88b32e3389054c6a9caf2e91ea917865 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/88b32e3389054c6a9caf2e91ea917865 2023-03-29 03:55:39,976 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/5e7b29bba65940acabd3c8d28c38897d to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/5e7b29bba65940acabd3c8d28c38897d 2023-03-29 03:55:39,977 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/002ddcbbb4c84fe58514457890b34e3d to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/002ddcbbb4c84fe58514457890b34e3d 2023-03-29 03:55:39,978 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/d0ed71e2627849f195cefc1eefb1ff72 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/d0ed71e2627849f195cefc1eefb1ff72 2023-03-29 03:55:39,978 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/e89c41ae679d43748bade17b8b1a1144 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/e89c41ae679d43748bade17b8b1a1144 2023-03-29 03:55:39,979 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/52b982e75b9a42089f729d3115dcfadd to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/52b982e75b9a42089f729d3115dcfadd 2023-03-29 03:55:39,980 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/15dee25bd8cd419c9c0971ac2d17662f to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/15dee25bd8cd419c9c0971ac2d17662f 2023-03-29 03:55:39,980 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/3285c402839d452589d36b939084301f to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/3285c402839d452589d36b939084301f 2023-03-29 03:55:39,981 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/fdd78514d3564102ba146d3c7df2cc02 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/fdd78514d3564102ba146d3c7df2cc02 2023-03-29 03:55:39,982 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/ee69bc58c4634710ac8e3827656bfc6b to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/ee69bc58c4634710ac8e3827656bfc6b 2023-03-29 03:55:39,982 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/08b2669d45bd4ab9b031f7107d833fbb to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/08b2669d45bd4ab9b031f7107d833fbb 2023-03-29 03:55:39,983 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/2f3277cda34c4a89a04edc2000b805cd to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/2f3277cda34c4a89a04edc2000b805cd 2023-03-29 03:55:39,984 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/72d73d4d55da4fd0a1bddb1f7fbf7dd9 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/72d73d4d55da4fd0a1bddb1f7fbf7dd9 2023-03-29 03:55:39,985 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/a2ceed1c461c495287a0cd19269dd47d to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/a2ceed1c461c495287a0cd19269dd47d 2023-03-29 03:55:39,985 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/e0056ddd6eed4fa9981d4d5e16a224aa to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/e0056ddd6eed4fa9981d4d5e16a224aa 2023-03-29 03:55:39,986 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/d081df086d5c4cce9a646c3c34c3e866 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/d081df086d5c4cce9a646c3c34c3e866 2023-03-29 03:55:39,987 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/76d8e08be1494c909ee3994baf678cd9 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/76d8e08be1494c909ee3994baf678cd9 2023-03-29 03:55:39,988 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/674edefba30a40b8a4c8013af97c971c to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/674edefba30a40b8a4c8013af97c971c 2023-03-29 03:55:39,988 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/7ccbcd2821f44f059c0458fa44806dbc to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/7ccbcd2821f44f059c0458fa44806dbc 2023-03-29 03:55:40,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-03-29 03:55:40,264 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=461 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/B/0ee4d99b1b02420c97e791e6cc8ce80c 2023-03-29 03:55:40,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-03-29 03:55:40,674 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=461 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/C/0e3c3f16933f43bc89b3d416c9f80593 2023-03-29 03:55:40,678 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/A/05a5587259004dadb5b6943828037619 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/05a5587259004dadb5b6943828037619 2023-03-29 03:55:40,681 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/05a5587259004dadb5b6943828037619, entries=150, sequenceid=461, filesize=25.4 K 2023-03-29 03:55:40,681 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/B/0ee4d99b1b02420c97e791e6cc8ce80c as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/0ee4d99b1b02420c97e791e6cc8ce80c 2023-03-29 03:55:40,684 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/0ee4d99b1b02420c97e791e6cc8ce80c, entries=150, sequenceid=461, filesize=11.9 K 2023-03-29 03:55:40,684 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/C/0e3c3f16933f43bc89b3d416c9f80593 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/0e3c3f16933f43bc89b3d416c9f80593 2023-03-29 03:55:40,687 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/0e3c3f16933f43bc89b3d416c9f80593, entries=150, sequenceid=461, filesize=11.9 K 2023-03-29 03:55:40,687 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~149.83 KB/153430, heapSize ~430.56 KB/440896, currentSize=20.13 KB/20610 for a5c78075cc0f4fd1864c2f0339ad3d67 in 1648ms, sequenceid=461, compaction requested=false 2023-03-29 03:55:40,687 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a5c78075cc0f4fd1864c2f0339ad3d67: 2023-03-29 03:55:40,687 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:40,687 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:40,687 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. after waiting 0 ms 2023-03-29 03:55:40,687 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:40,687 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing a5c78075cc0f4fd1864c2f0339ad3d67 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-03-29 03:55:40,688 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5c78075cc0f4fd1864c2f0339ad3d67, store=A 2023-03-29 03:55:40,688 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:55:40,688 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5c78075cc0f4fd1864c2f0339ad3d67, store=B 2023-03-29 03:55:40,688 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:55:40,688 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a5c78075cc0f4fd1864c2f0339ad3d67, store=C 2023-03-29 03:55:40,688 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:55:40,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-03-29 03:55:41,096 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303296a5370246e3c42c8a34f81e87e1aae87 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303296a5370246e3c42c8a34f81e87e1aae87 2023-03-29 03:55:41,501 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=470, memsize=6.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/A/6c197b6ec8f648f684ec8ff266753de3 2023-03-29 03:55:41,508 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=470 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/B/298221c1e63a4e47ba3ba7739c3b7a44 2023-03-29 03:55:41,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-03-29 03:55:41,915 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=470 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/C/e96fc955d6a8425e8914e334ebda1a12 2023-03-29 03:55:41,919 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/A/6c197b6ec8f648f684ec8ff266753de3 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/6c197b6ec8f648f684ec8ff266753de3 2023-03-29 03:55:41,922 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/6c197b6ec8f648f684ec8ff266753de3, entries=100, sequenceid=470, filesize=18.6 K 2023-03-29 03:55:41,923 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/B/298221c1e63a4e47ba3ba7739c3b7a44 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/298221c1e63a4e47ba3ba7739c3b7a44 2023-03-29 03:55:41,926 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/298221c1e63a4e47ba3ba7739c3b7a44, entries=100, sequenceid=470, filesize=9.5 K 2023-03-29 03:55:41,927 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/.tmp/C/e96fc955d6a8425e8914e334ebda1a12 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/e96fc955d6a8425e8914e334ebda1a12 2023-03-29 03:55:41,930 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/e96fc955d6a8425e8914e334ebda1a12, entries=100, sequenceid=470, filesize=9.5 K 2023-03-29 03:55:41,931 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.13 KB/20610, heapSize ~60.47 KB/61920, currentSize=0 B/0 for a5c78075cc0f4fd1864c2f0339ad3d67 in 1243ms, sequenceid=470, compaction requested=true 2023-03-29 03:55:41,934 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/recovered.edits/473.seqid, newMaxSeqId=473, maxSeqId=4 2023-03-29 03:55:41,935 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67. 2023-03-29 03:55:41,935 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for a5c78075cc0f4fd1864c2f0339ad3d67: 2023-03-29 03:55:41,937 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed a5c78075cc0f4fd1864c2f0339ad3d67 2023-03-29 03:55:41,937 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=44 updating hbase:meta row=a5c78075cc0f4fd1864c2f0339ad3d67, regionState=CLOSED 2023-03-29 03:55:41,937 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680062141937"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680062141937"}]},"ts":"1680062141937"} 2023-03-29 03:55:41,939 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=45, resume processing ppid=44 2023-03-29 03:55:41,939 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=45, ppid=44, state=SUCCESS; CloseRegionProcedure a5c78075cc0f4fd1864c2f0339ad3d67, server=jenkins-hbase4.apache.org,35599,1680062018534 in 2.1210 sec 2023-03-29 03:55:41,940 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=44, resume processing ppid=43 2023-03-29 03:55:41,940 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=44, ppid=43, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=a5c78075cc0f4fd1864c2f0339ad3d67, UNASSIGN in 2.1250 sec 2023-03-29 03:55:41,941 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680062141941"}]},"ts":"1680062141941"} 2023-03-29 03:55:41,942 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-03-29 03:55:41,943 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-03-29 03:55:41,945 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=43, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 2.2320 sec 2023-03-29 03:55:43,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-03-29 03:55:43,821 INFO [Listener at localhost/35027] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 43 completed 2023-03-29 03:55:43,822 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-03-29 03:55:43,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure2.ProcedureExecutor(1029): Stored pid=46, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-03-29 03:55:43,824 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=46, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-29 03:55:43,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=46 2023-03-29 03:55:43,824 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=46, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-29 03:55:43,827 DEBUG [HFileArchiver-9] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/.tmp/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67 2023-03-29 03:55:43,829 DEBUG [HFileArchiver-9] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/.tmp/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A, FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/.tmp/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B, FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/.tmp/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C, FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/.tmp/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/recovered.edits] 2023-03-29 03:55:43,831 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/.tmp/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/05a5587259004dadb5b6943828037619 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/05a5587259004dadb5b6943828037619 2023-03-29 03:55:43,832 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/.tmp/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/6c197b6ec8f648f684ec8ff266753de3 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/6c197b6ec8f648f684ec8ff266753de3 2023-03-29 03:55:43,833 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/.tmp/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/bbf8e1beffc540a89c62d402ed7ac930 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/A/bbf8e1beffc540a89c62d402ed7ac930 2023-03-29 03:55:43,835 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/.tmp/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/0ee4d99b1b02420c97e791e6cc8ce80c to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/0ee4d99b1b02420c97e791e6cc8ce80c 2023-03-29 03:55:43,836 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/.tmp/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/298221c1e63a4e47ba3ba7739c3b7a44 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/298221c1e63a4e47ba3ba7739c3b7a44 2023-03-29 03:55:43,837 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/.tmp/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/6ca81f3168ce4f368d5963cc7ba52510 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/B/6ca81f3168ce4f368d5963cc7ba52510 2023-03-29 03:55:43,838 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/.tmp/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/0e3c3f16933f43bc89b3d416c9f80593 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/0e3c3f16933f43bc89b3d416c9f80593 2023-03-29 03:55:43,839 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/.tmp/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/527f0f58cd2840e7b15292e60d5c695a to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/527f0f58cd2840e7b15292e60d5c695a 2023-03-29 03:55:43,840 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/.tmp/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/e96fc955d6a8425e8914e334ebda1a12 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/C/e96fc955d6a8425e8914e334ebda1a12 2023-03-29 03:55:43,842 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/.tmp/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/recovered.edits/473.seqid to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67/recovered.edits/473.seqid 2023-03-29 03:55:43,842 DEBUG [HFileArchiver-9] backup.HFileArchiver(596): Deleted hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/.tmp/data/default/TestAcidGuarantees/a5c78075cc0f4fd1864c2f0339ad3d67 2023-03-29 03:55:43,842 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-29 03:55:43,843 DEBUG [PEWorker-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-03-29 03:55:43,843 DEBUG [PEWorker-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-03-29 03:55:43,845 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032903b1f440c7874353a58fd17b635513ee to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032903b1f440c7874353a58fd17b635513ee 2023-03-29 03:55:43,846 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303292346cc3fd25d4e9a8f999affccf42486 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303292346cc3fd25d4e9a8f999affccf42486 2023-03-29 03:55:43,847 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303292567b7aa0f2144e9b16fc0ec4c384f65 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303292567b7aa0f2144e9b16fc0ec4c384f65 2023-03-29 03:55:43,847 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303292d3df0d78b1847d58b0d5e3c9587a7fe to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303292d3df0d78b1847d58b0d5e3c9587a7fe 2023-03-29 03:55:43,848 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032940d4169d8e10421785da2f748d2b4217 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032940d4169d8e10421785da2f748d2b4217 2023-03-29 03:55:43,849 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230329433df01ad1984d13a4a865d8ebc4a19e to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230329433df01ad1984d13a4a865d8ebc4a19e 2023-03-29 03:55:43,850 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032963af062249ca48cbb8704fd4236e3376 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032963af062249ca48cbb8704fd4236e3376 2023-03-29 03:55:43,851 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303296460d796f8e945c78aa3e9c71a325a68 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303296460d796f8e945c78aa3e9c71a325a68 2023-03-29 03:55:43,851 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303296a5370246e3c42c8a34f81e87e1aae87 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303296a5370246e3c42c8a34f81e87e1aae87 2023-03-29 03:55:43,852 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032987b62070eed243d7944dd28f8e2f2cff to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032987b62070eed243d7944dd28f8e2f2cff 2023-03-29 03:55:43,853 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032988e1d2764d8a4466965d4a7f5df539ef to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032988e1d2764d8a4466965d4a7f5df539ef 2023-03-29 03:55:43,854 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303298e9008f20e5940f5b89e2bf5752624c0 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303298e9008f20e5940f5b89e2bf5752624c0 2023-03-29 03:55:43,855 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032993ea0204be1343bcb84df175f68f7613 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032993ea0204be1343bcb84df175f68f7613 2023-03-29 03:55:43,856 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303299e5576e0b93c422bb557a58db2a90df6 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303299e5576e0b93c422bb557a58db2a90df6 2023-03-29 03:55:43,857 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230329a156ea92d05d486ca03c7f37b9e8be2c to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230329a156ea92d05d486ca03c7f37b9e8be2c 2023-03-29 03:55:43,857 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230329b68f687e27734966b85d725b421c9fe1 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230329b68f687e27734966b85d725b421c9fe1 2023-03-29 03:55:43,859 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230329bb625559ac2e4d17b6ff10f0850b47d1 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230329bb625559ac2e4d17b6ff10f0850b47d1 2023-03-29 03:55:43,860 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230329d46615661c264b1e80b4113fe1427c1d to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230329d46615661c264b1e80b4113fe1427c1d 2023-03-29 03:55:43,861 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230329f1cd578ad8d24ea0aba16fc90c0c7265 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230329f1cd578ad8d24ea0aba16fc90c0c7265 2023-03-29 03:55:43,862 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230329f1f5b27850674fb8bf77f02be0ae1fb1 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230329f1f5b27850674fb8bf77f02be0ae1fb1 2023-03-29 03:55:43,863 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230329fe3c99e3fb914c16beee4e4f697614a4 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230329fe3c99e3fb914c16beee4e4f697614a4 2023-03-29 03:55:43,864 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230329fef9462da461456bba94701a93985194 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230329fef9462da461456bba94701a93985194 2023-03-29 03:55:43,865 DEBUG [PEWorker-3] backup.HFileArchiver(596): Deleted hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-03-29 03:55:43,867 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=46, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-29 03:55:43,868 WARN [PEWorker-3] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-03-29 03:55:43,869 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-03-29 03:55:43,870 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=46, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-29 03:55:43,870 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-03-29 03:55:43,870 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1680062143870"}]},"ts":"9223372036854775807"} 2023-03-29 03:55:43,872 INFO [PEWorker-3] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-03-29 03:55:43,872 DEBUG [PEWorker-3] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => a5c78075cc0f4fd1864c2f0339ad3d67, NAME => 'TestAcidGuarantees,,1680062115799.a5c78075cc0f4fd1864c2f0339ad3d67.', STARTKEY => '', ENDKEY => ''}] 2023-03-29 03:55:43,872 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-03-29 03:55:43,872 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1680062143872"}]},"ts":"9223372036854775807"} 2023-03-29 03:55:43,873 INFO [PEWorker-3] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-03-29 03:55:43,874 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(130): Finished pid=46, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-29 03:55:43,875 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=46, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 53 msec 2023-03-29 03:55:43,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=46 2023-03-29 03:55:43,925 INFO [Listener at localhost/35027] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 46 completed 2023-03-29 03:55:43,936 INFO [Listener at localhost/35027] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobScanAtomicity Thread=234 (was 224) - Thread LEAK? -, OpenFileDescriptor=560 (was 513) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=419 (was 367) - SystemLoadAverage LEAK? -, ProcessCount=169 (was 169), AvailableMemoryMB=6595 (was 6633) 2023-03-29 03:55:43,948 INFO [Listener at localhost/35027] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testScanAtomicity Thread=234, OpenFileDescriptor=560, MaxFileDescriptor=60000, SystemLoadAverage=419, ProcessCount=171, AvailableMemoryMB=6594 2023-03-29 03:55:43,950 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-29 03:55:43,950 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-29 03:55:43,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure2.ProcedureExecutor(1029): Stored pid=47, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-03-29 03:55:43,952 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-03-29 03:55:43,952 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 47 2023-03-29 03:55:43,953 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-03-29 03:55:43,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-03-29 03:55:43,954 DEBUG [HFileArchiver-10] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/.tmp/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758 2023-03-29 03:55:43,954 DEBUG [HFileArchiver-10] backup.HFileArchiver(153): Directory hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/.tmp/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758 empty. 2023-03-29 03:55:43,954 DEBUG [HFileArchiver-10] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/.tmp/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758 2023-03-29 03:55:43,954 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-29 03:55:44,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-03-29 03:55:44,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-03-29 03:55:44,361 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-29 03:55:44,363 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 0337be0f939be2a2b593eefecee32758, NAME => 'TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/.tmp 2023-03-29 03:55:44,370 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-29 03:55:44,370 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 0337be0f939be2a2b593eefecee32758, disabling compactions & flushes 2023-03-29 03:55:44,370 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:44,370 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:44,370 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. after waiting 0 ms 2023-03-29 03:55:44,371 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:44,371 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:44,371 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 0337be0f939be2a2b593eefecee32758: 2023-03-29 03:55:44,373 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-03-29 03:55:44,374 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680062144373"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680062144373"}]},"ts":"1680062144373"} 2023-03-29 03:55:44,374 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-03-29 03:55:44,375 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-03-29 03:55:44,375 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680062144375"}]},"ts":"1680062144375"} 2023-03-29 03:55:44,376 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-03-29 03:55:44,379 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=0337be0f939be2a2b593eefecee32758, ASSIGN}] 2023-03-29 03:55:44,380 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=0337be0f939be2a2b593eefecee32758, ASSIGN 2023-03-29 03:55:44,381 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=0337be0f939be2a2b593eefecee32758, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,35599,1680062018534; forceNewPlan=false, retain=false 2023-03-29 03:55:44,532 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=48 updating hbase:meta row=0337be0f939be2a2b593eefecee32758, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:44,532 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680062144532"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680062144532"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680062144532"}]},"ts":"1680062144532"} 2023-03-29 03:55:44,533 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=49, ppid=48, state=RUNNABLE; OpenRegionProcedure 0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534}] 2023-03-29 03:55:44,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-03-29 03:55:44,687 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:44,687 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 0337be0f939be2a2b593eefecee32758, NAME => 'TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.', STARTKEY => '', ENDKEY => ''} 2023-03-29 03:55:44,687 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 0337be0f939be2a2b593eefecee32758 2023-03-29 03:55:44,688 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-29 03:55:44,688 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 0337be0f939be2a2b593eefecee32758 2023-03-29 03:55:44,688 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 0337be0f939be2a2b593eefecee32758 2023-03-29 03:55:44,689 INFO [StoreOpener-0337be0f939be2a2b593eefecee32758-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 0337be0f939be2a2b593eefecee32758 2023-03-29 03:55:44,690 DEBUG [StoreOpener-0337be0f939be2a2b593eefecee32758-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A 2023-03-29 03:55:44,690 DEBUG [StoreOpener-0337be0f939be2a2b593eefecee32758-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A 2023-03-29 03:55:44,691 INFO [StoreOpener-0337be0f939be2a2b593eefecee32758-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-29 03:55:44,691 INFO [StoreOpener-0337be0f939be2a2b593eefecee32758-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 0337be0f939be2a2b593eefecee32758 columnFamilyName A 2023-03-29 03:55:44,692 INFO [StoreOpener-0337be0f939be2a2b593eefecee32758-1] regionserver.HStore(310): Store=0337be0f939be2a2b593eefecee32758/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-29 03:55:44,692 INFO [StoreOpener-0337be0f939be2a2b593eefecee32758-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 0337be0f939be2a2b593eefecee32758 2023-03-29 03:55:44,693 DEBUG [StoreOpener-0337be0f939be2a2b593eefecee32758-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B 2023-03-29 03:55:44,694 DEBUG [StoreOpener-0337be0f939be2a2b593eefecee32758-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B 2023-03-29 03:55:44,694 INFO [StoreOpener-0337be0f939be2a2b593eefecee32758-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-29 03:55:44,694 INFO [StoreOpener-0337be0f939be2a2b593eefecee32758-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 0337be0f939be2a2b593eefecee32758 columnFamilyName B 2023-03-29 03:55:44,694 INFO [StoreOpener-0337be0f939be2a2b593eefecee32758-1] regionserver.HStore(310): Store=0337be0f939be2a2b593eefecee32758/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-29 03:55:44,695 INFO [StoreOpener-0337be0f939be2a2b593eefecee32758-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 0337be0f939be2a2b593eefecee32758 2023-03-29 03:55:44,696 DEBUG [StoreOpener-0337be0f939be2a2b593eefecee32758-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C 2023-03-29 03:55:44,696 DEBUG [StoreOpener-0337be0f939be2a2b593eefecee32758-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C 2023-03-29 03:55:44,696 INFO [StoreOpener-0337be0f939be2a2b593eefecee32758-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-29 03:55:44,696 INFO [StoreOpener-0337be0f939be2a2b593eefecee32758-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 0337be0f939be2a2b593eefecee32758 columnFamilyName C 2023-03-29 03:55:44,697 INFO [StoreOpener-0337be0f939be2a2b593eefecee32758-1] regionserver.HStore(310): Store=0337be0f939be2a2b593eefecee32758/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-29 03:55:44,697 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:44,697 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758 2023-03-29 03:55:44,697 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758 2023-03-29 03:55:44,699 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-29 03:55:44,700 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 0337be0f939be2a2b593eefecee32758 2023-03-29 03:55:44,702 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-29 03:55:44,702 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 0337be0f939be2a2b593eefecee32758; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73977335, jitterRate=0.102348193526268}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-29 03:55:44,702 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 0337be0f939be2a2b593eefecee32758: 2023-03-29 03:55:44,703 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758., pid=49, masterSystemTime=1680062144685 2023-03-29 03:55:44,704 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:44,704 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:44,704 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=48 updating hbase:meta row=0337be0f939be2a2b593eefecee32758, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:44,704 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680062144704"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1680062144704"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680062144704"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680062144704"}]},"ts":"1680062144704"} 2023-03-29 03:55:44,706 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=49, resume processing ppid=48 2023-03-29 03:55:44,706 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=49, ppid=48, state=SUCCESS; OpenRegionProcedure 0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 in 172 msec 2023-03-29 03:55:44,708 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=48, resume processing ppid=47 2023-03-29 03:55:44,708 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=48, ppid=47, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=0337be0f939be2a2b593eefecee32758, ASSIGN in 327 msec 2023-03-29 03:55:44,708 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-03-29 03:55:44,708 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680062144708"}]},"ts":"1680062144708"} 2023-03-29 03:55:44,709 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-03-29 03:55:44,712 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-03-29 03:55:44,713 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=47, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 762 msec 2023-03-29 03:55:45,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-03-29 03:55:45,056 INFO [Listener at localhost/35027] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 47 completed 2023-03-29 03:55:45,057 DEBUG [Listener at localhost/35027] zookeeper.ReadOnlyZKClient(139): Connect 0x34ad4aa8 to 127.0.0.1:52103 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-29 03:55:45,062 DEBUG [Listener at localhost/35027] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@52203cb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-29 03:55:45,063 DEBUG [Listener at localhost/35027] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-29 03:55:45,065 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41940, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-29 03:55:45,066 DEBUG [Listener at localhost/35027] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-03-29 03:55:45,067 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:44532, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-03-29 03:55:45,068 DEBUG [Listener at localhost/35027] zookeeper.ReadOnlyZKClient(139): Connect 0x5726fb8b to 127.0.0.1:52103 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-29 03:55:45,075 DEBUG [Listener at localhost/35027] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6edd017, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-29 03:55:45,076 DEBUG [Listener at localhost/35027] zookeeper.ReadOnlyZKClient(139): Connect 0x6f2b3c03 to 127.0.0.1:52103 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-29 03:55:45,081 DEBUG [Listener at localhost/35027] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@fb168ce, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-29 03:55:45,081 DEBUG [Listener at localhost/35027] zookeeper.ReadOnlyZKClient(139): Connect 0x70e2be04 to 127.0.0.1:52103 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-29 03:55:45,085 DEBUG [Listener at localhost/35027] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@461f5f3f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-29 03:55:45,085 DEBUG [Listener at localhost/35027] zookeeper.ReadOnlyZKClient(139): Connect 0x067ea274 to 127.0.0.1:52103 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-29 03:55:45,089 DEBUG [Listener at localhost/35027] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@63093e21, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-29 03:55:45,089 DEBUG [Listener at localhost/35027] zookeeper.ReadOnlyZKClient(139): Connect 0x78592df0 to 127.0.0.1:52103 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-29 03:55:45,093 DEBUG [Listener at localhost/35027] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@b475d8d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-29 03:55:45,093 DEBUG [Listener at localhost/35027] zookeeper.ReadOnlyZKClient(139): Connect 0x549bef45 to 127.0.0.1:52103 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-29 03:55:45,097 DEBUG [Listener at localhost/35027] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@65816b6e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-29 03:55:45,097 DEBUG [Listener at localhost/35027] zookeeper.ReadOnlyZKClient(139): Connect 0x6ed183a4 to 127.0.0.1:52103 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-29 03:55:45,102 DEBUG [Listener at localhost/35027] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@653bb8fc, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-29 03:55:45,103 DEBUG [Listener at localhost/35027] zookeeper.ReadOnlyZKClient(139): Connect 0x10a344e6 to 127.0.0.1:52103 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-29 03:55:45,106 DEBUG [Listener at localhost/35027] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@720d29f5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-29 03:55:45,107 DEBUG [Listener at localhost/35027] zookeeper.ReadOnlyZKClient(139): Connect 0x3a9f3972 to 127.0.0.1:52103 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-29 03:55:45,116 DEBUG [Listener at localhost/35027] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3011deb3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-29 03:55:45,116 DEBUG [Listener at localhost/35027] zookeeper.ReadOnlyZKClient(139): Connect 0x2d9fccbf to 127.0.0.1:52103 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-29 03:55:45,120 DEBUG [Listener at localhost/35027] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6e8824d7, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-29 03:55:45,124 DEBUG [hconnection-0x4021e79d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-29 03:55:45,124 DEBUG [hconnection-0x52ac768e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-29 03:55:45,126 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:45,126 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41946, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-29 03:55:45,127 DEBUG [hconnection-0x43d0b66-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-29 03:55:45,127 DEBUG [hconnection-0x46e41eef-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-29 03:55:45,127 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41942, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-29 03:55:45,127 DEBUG [hconnection-0x380e455a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-29 03:55:45,128 DEBUG [hconnection-0x580f8899-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-29 03:55:45,128 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41990, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-29 03:55:45,131 DEBUG [hconnection-0x1f4fc121-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-29 03:55:45,132 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41960, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-29 03:55:45,132 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41976, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-29 03:55:45,132 DEBUG [hconnection-0x6a4c1757-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-29 03:55:45,133 DEBUG [hconnection-0xfa5973-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-29 03:55:45,133 DEBUG [hconnection-0x70cf3025-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-29 03:55:45,136 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:42004, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-29 03:55:45,136 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:42008, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-29 03:55:45,140 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:42010, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-29 03:55:45,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:45,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:45,141 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:42012, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-29 03:55:45,142 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:45,145 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:45,145 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:45,145 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:45,146 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:45,146 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:45,148 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:42024, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-29 03:55:45,148 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:45,149 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:45,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:45,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:45,149 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:45,149 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:45,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:45,151 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:45,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:45,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:45,152 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:45,153 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:45,153 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:45,153 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:45,153 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:45,153 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:45,153 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:45,153 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:45,153 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:45,154 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. started... 2023-03-29 03:55:45,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on 0337be0f939be2a2b593eefecee32758 2023-03-29 03:55:45,154 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 0337be0f939be2a2b593eefecee32758 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-29 03:55:45,154 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. as already flushing 2023-03-29 03:55:45,154 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:45,154 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:45,154 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:45,154 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:45,154 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:45,154 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:45,155 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0337be0f939be2a2b593eefecee32758, store=A 2023-03-29 03:55:45,155 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:55:45,155 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0337be0f939be2a2b593eefecee32758, store=B 2023-03-29 03:55:45,155 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:55:45,155 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0337be0f939be2a2b593eefecee32758, store=C 2023-03-29 03:55:45,155 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:55:45,155 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:45,155 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:45,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:45,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:45,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:45,156 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:45,156 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:45,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:45,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:45,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:45,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:45,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:45,160 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:45,161 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@46e3ecd9[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:45,161 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:45,161 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:45,163 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:45,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:45,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:45,163 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:45,163 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:45,163 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:45,163 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:45,163 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:45,164 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:45,164 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:45,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:45,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:45,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:45,164 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:45,165 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:45,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:45,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:45,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:45,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:45,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:45,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:45,169 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:45,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:45,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41960 deadline: 1680062205166, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:45,169 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:45,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42004 deadline: 1680062205166, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:45,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:45,170 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:45,170 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:45,170 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:45,170 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:45,170 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:45,170 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:45,172 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:45,172 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:45,172 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:45,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:45,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:45,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:45,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:45,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:45,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:45,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:45,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:45,172 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:45,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:45,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:45,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:45,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:45,173 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:45,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:45,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42010 deadline: 1680062205167, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:45,174 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:45,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:45,174 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:45,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41942 deadline: 1680062205168, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:45,174 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:45,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:45,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42024 deadline: 1680062205169, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:45,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:45,176 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:45,176 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:45,176 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:45,176 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:45,176 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:45,176 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:45,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:45,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:45,176 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:45,176 DEBUG [Thread-3261] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:45,176 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:45,176 DEBUG [Thread-3261] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:45,176 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:45,177 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:45,177 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:45,177 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=14 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/A/864bc06b60e74a85addefd04b73687ac 2023-03-29 03:55:45,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:45,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:45,177 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:45,273 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:45,273 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:45,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41960 deadline: 1680062205270, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:45,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42004 deadline: 1680062205270, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:45,277 DEBUG [Thread-3261] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:45,277 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:45,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:45,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42010 deadline: 1680062205275, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:45,279 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:45,280 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:45,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41942 deadline: 1680062205275, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:45,280 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:45,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42024 deadline: 1680062205275, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:45,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:45,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:45,285 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:45,285 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:45,285 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:45,285 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:45,285 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:45,285 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:45,287 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:45,287 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:45,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:45,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:45,287 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:45,287 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:45,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:45,288 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:45,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:45,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:45,288 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:45,292 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:45,292 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:45,292 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:45,292 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:45,292 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:45,292 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:45,293 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:45,300 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:45,300 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. started... 2023-03-29 03:55:45,300 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. as already flushing 2023-03-29 03:55:45,300 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:45,300 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:45,300 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:45,300 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:45,300 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:45,300 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:45,302 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:45,302 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:45,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:45,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:45,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:45,303 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:45,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:45,303 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:45,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:45,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:45,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:45,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:45,304 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:45,304 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@78d53600[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:45,304 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:45,304 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:45,306 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:45,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:45,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:45,306 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:45,306 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:45,306 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:45,306 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:45,306 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:45,312 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:45,312 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:45,312 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:45,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:45,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:45,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:45,312 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:45,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:45,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:45,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:45,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:45,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:45,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:45,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:45,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:45,314 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:45,315 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:45,315 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:45,315 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:45,315 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:45,315 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:45,316 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:45,316 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:45,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:45,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:45,320 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:45,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:45,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:45,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:45,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:45,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:45,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:45,320 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:45,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:45,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:45,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:45,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:45,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:45,322 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:45,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:45,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:45,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:45,324 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:45,324 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:45,324 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:45,324 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:45,324 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:45,324 DEBUG [Thread-3261] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:45,324 DEBUG [Thread-3261] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:45,324 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:45,325 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:45,324 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:45,325 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:45,325 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:45,325 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:45,325 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:45,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:45,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:45,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:45,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:45,425 DEBUG [Thread-3261] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:45,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:45,427 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:45,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:45,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:45,428 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:45,428 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:45,428 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:45,428 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:45,428 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:45,428 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:45,430 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:45,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:45,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:45,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:45,431 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:45,431 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:45,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:45,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:45,431 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:45,432 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:45,432 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:45,432 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:45,432 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:45,432 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:45,432 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:45,432 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:45,438 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:45,438 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:45,438 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:45,438 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. started... 2023-03-29 03:55:45,438 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. as already flushing 2023-03-29 03:55:45,438 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:45,438 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:45,438 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:45,438 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:45,438 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:45,438 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:45,440 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:45,440 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:45,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:45,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:45,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:45,440 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:45,440 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:45,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:45,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:45,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:45,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:45,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:45,442 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:45,442 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3570b689[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:45,442 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:45,442 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:45,444 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:45,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:45,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:45,444 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:45,444 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:45,444 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:45,446 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:45,446 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:45,446 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:45,446 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:45,447 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:45,447 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:45,447 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:45,448 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:45,448 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:45,448 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:45,448 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:45,451 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:45,451 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:45,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:45,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:45,451 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:45,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:45,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:45,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:45,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:45,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:45,452 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:45,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:45,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:45,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:45,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:45,453 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:45,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:45,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:45,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:45,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:45,458 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:45,458 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:45,458 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:45,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:45,458 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:45,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:45,458 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:45,458 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:55:45,458 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:45,458 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:45,458 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:45,458 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:45,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:45,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:45,458 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:45,458 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:45,458 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:45,458 DEBUG [Thread-3261] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:45,458 DEBUG [Thread-3261] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:45,477 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:45,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42004 deadline: 1680062205474, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:45,478 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:45,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41960 deadline: 1680062205475, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:45,482 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:45,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42010 deadline: 1680062205479, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:45,484 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:45,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42024 deadline: 1680062205481, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:45,484 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:45,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41942 deadline: 1680062205482, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:45,559 DEBUG [Thread-3261] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:45,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:45,561 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:45,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:45,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:45,561 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:45,562 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:45,562 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:45,562 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:45,562 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:45,562 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:45,563 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:45,563 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:45,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:45,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:45,563 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:45,564 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:45,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:45,564 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:45,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:45,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:45,564 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:45,565 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:45,565 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:45,565 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:45,565 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:45,565 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:45,565 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:45,565 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:45,565 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:45,565 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. started... 2023-03-29 03:55:45,565 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. as already flushing 2023-03-29 03:55:45,565 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:45,565 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:45,565 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:45,565 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:45,565 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:45,565 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:45,568 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:45,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:45,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:45,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:45,568 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:45,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:45,569 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:45,569 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:45,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:45,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:45,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:45,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:45,570 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:45,570 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@74c3822b[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:45,570 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:45,570 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:45,571 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:45,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:45,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:45,571 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:45,571 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:45,571 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:45,571 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:45,571 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:45,573 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:45,573 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:45,573 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:45,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:45,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:45,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:45,573 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:45,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:45,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:45,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:45,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:45,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:45,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:45,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:45,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:45,575 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:45,575 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:45,575 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:45,575 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:45,575 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:45,575 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:45,577 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:45,577 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:45,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:45,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:45,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:45,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:45,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:45,577 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:45,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:45,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:45,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:45,577 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:45,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:45,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:45,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:45,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:45,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:45,578 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:45,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:45,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:45,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:45,581 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:45,581 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:45,581 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:45,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:45,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:45,581 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:45,582 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:45,582 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:45,582 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:45,581 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:45,582 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:45,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:45,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:45,582 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:45,582 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:45,582 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:45,583 DEBUG [Thread-3261] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:45,583 DEBUG [Thread-3261] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:45,592 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=14 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/B/2cda48bbe6884a9ebddda4f078f5687d 2023-03-29 03:55:45,683 DEBUG [Thread-3261] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:45,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:45,685 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:45,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:45,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:45,686 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:45,686 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:45,686 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:45,686 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:45,687 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:45,687 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:45,688 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:45,688 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:45,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:45,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:45,688 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:45,688 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:45,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:45,689 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:45,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:45,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:45,689 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:45,689 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:45,689 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:45,689 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:45,690 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:45,690 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:45,690 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:45,690 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:45,690 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:45,690 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. started... 2023-03-29 03:55:45,690 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. as already flushing 2023-03-29 03:55:45,690 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:45,690 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:45,690 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:45,690 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:45,690 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:45,690 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:45,692 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:45,692 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:45,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:45,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:45,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:45,692 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:45,692 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:45,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:45,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:45,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:45,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:45,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:45,693 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:45,693 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d9f3d4e[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:45,693 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:45,694 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:45,695 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:45,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:45,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:45,695 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:45,695 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:45,695 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:45,696 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:45,696 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:45,698 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:45,698 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:45,698 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:45,698 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:45,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:45,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:45,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:45,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:45,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:45,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:45,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:45,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:45,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:45,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:45,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:45,700 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:45,700 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:45,700 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:45,700 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:45,700 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:45,700 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:45,702 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:45,702 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:45,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:45,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:45,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:45,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:45,702 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:45,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:45,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:45,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:45,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:45,702 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:45,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:45,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:45,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:45,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:45,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:45,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:45,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:45,706 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:45,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:45,708 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:45,708 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:45,708 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:45,708 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:45,708 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:45,708 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:45,708 DEBUG [Thread-3261] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:45,708 DEBUG [Thread-3261] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:45,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:45,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:45,708 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:45,708 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:45,709 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:45,709 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:45,709 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:45,709 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:45,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:45,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:45,781 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:45,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42004 deadline: 1680062205780, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:45,782 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:45,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41960 deadline: 1680062205781, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:45,785 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:45,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42010 deadline: 1680062205783, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:45,788 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:45,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42024 deadline: 1680062205786, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:45,789 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:45,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41942 deadline: 1680062205787, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:45,809 DEBUG [Thread-3261] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:45,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:45,811 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:45,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:45,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:45,812 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:45,812 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:45,812 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:45,812 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:45,812 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:45,812 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:45,814 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:45,814 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:45,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:45,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:45,815 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:45,815 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:45,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:45,815 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:45,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:45,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:45,816 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:45,817 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:45,817 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:45,817 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:45,817 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:45,817 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:45,818 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:45,818 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:45,818 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:45,818 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. started... 2023-03-29 03:55:45,818 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. as already flushing 2023-03-29 03:55:45,818 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:45,818 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:45,818 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:45,818 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:45,818 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:45,818 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:45,820 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:45,820 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:45,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:45,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:45,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:45,821 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:45,821 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:45,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:45,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:45,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:45,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:45,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:45,823 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:45,823 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7cab6bcb[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:45,823 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:45,824 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:45,825 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:45,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:45,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:45,825 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:45,825 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:45,825 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:45,825 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:45,825 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:45,827 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:45,827 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:45,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:45,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:45,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:45,827 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:45,827 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:45,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:45,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:45,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:45,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:45,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:45,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:45,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:45,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:45,830 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:45,830 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:45,830 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:45,830 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:45,830 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:45,830 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:45,831 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:45,832 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:45,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:45,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:45,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:45,832 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:45,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:45,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:45,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:45,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:45,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:45,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:45,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:45,834 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:45,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:45,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:45,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:45,835 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:45,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:45,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:45,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:45,838 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:45,838 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:45,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:45,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:45,838 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:45,838 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:45,839 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:45,839 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:45,839 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:45,839 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:45,839 DEBUG [Thread-3261] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:45,839 DEBUG [Thread-3261] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:45,839 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:45,839 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:45,839 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:45,839 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:45,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:45,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:45,939 DEBUG [Thread-3261] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:45,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:45,941 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:45,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:45,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:45,943 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:45,943 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:45,943 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:45,943 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:45,943 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:45,943 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:45,947 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:45,947 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:45,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:45,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:45,948 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:45,948 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:45,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:45,948 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:45,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:45,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:45,948 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:45,949 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:45,949 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:45,949 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:45,949 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:45,949 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:45,949 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:45,950 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:45,950 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:45,950 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. started... 2023-03-29 03:55:45,950 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. as already flushing 2023-03-29 03:55:45,950 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:45,950 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:45,950 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:45,950 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:45,950 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:45,950 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:45,951 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:45,951 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:45,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:45,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:45,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:45,952 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:45,952 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:45,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:45,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:45,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:45,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:45,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:45,953 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:45,953 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c5f5e29[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:45,953 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:45,953 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:45,955 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:45,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:45,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:45,955 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:45,955 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:45,955 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:45,955 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:45,955 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:45,957 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:45,957 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:45,957 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:45,957 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:45,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:45,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:45,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:45,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:45,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:45,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:45,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:45,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:45,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:45,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:45,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:45,959 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:45,959 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:45,959 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:45,959 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:45,959 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:45,959 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:45,961 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:45,961 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:45,961 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:45,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:45,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:45,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:45,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:45,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:45,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:45,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:45,961 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:45,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:45,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:45,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:45,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:45,963 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:45,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:45,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:45,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:45,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:45,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:45,965 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:45,965 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:45,965 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:45,965 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:45,965 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:45,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:45,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:45,965 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:45,965 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:55:45,965 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:45,966 DEBUG [Thread-3261] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:45,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:45,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:45,966 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:45,966 DEBUG [Thread-3261] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:45,966 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:45,966 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:45,966 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:45,966 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:46,001 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=14 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/C/ff07ce7ad8024f97a6be2f47b92a3b03 2023-03-29 03:55:46,008 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/A/864bc06b60e74a85addefd04b73687ac as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/864bc06b60e74a85addefd04b73687ac 2023-03-29 03:55:46,015 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/864bc06b60e74a85addefd04b73687ac, entries=150, sequenceid=14, filesize=11.6 K 2023-03-29 03:55:46,016 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/B/2cda48bbe6884a9ebddda4f078f5687d as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/2cda48bbe6884a9ebddda4f078f5687d 2023-03-29 03:55:46,019 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/2cda48bbe6884a9ebddda4f078f5687d, entries=150, sequenceid=14, filesize=11.6 K 2023-03-29 03:55:46,022 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/C/ff07ce7ad8024f97a6be2f47b92a3b03 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/ff07ce7ad8024f97a6be2f47b92a3b03 2023-03-29 03:55:46,025 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/ff07ce7ad8024f97a6be2f47b92a3b03, entries=150, sequenceid=14, filesize=11.6 K 2023-03-29 03:55:46,026 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 0337be0f939be2a2b593eefecee32758 in 872ms, sequenceid=14, compaction requested=false 2023-03-29 03:55:46,026 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 0337be0f939be2a2b593eefecee32758: 2023-03-29 03:55:46,066 DEBUG [Thread-3261] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:46,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:46,068 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:46,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:46,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:46,069 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:46,069 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:46,069 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:46,069 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:46,070 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:46,070 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:46,072 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:46,072 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:46,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:46,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:46,073 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:46,073 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:46,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:46,073 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:46,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:46,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:46,073 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:46,074 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:46,074 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:46,074 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:46,074 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:46,074 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:46,074 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:46,074 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:46,074 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:46,074 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. started... 2023-03-29 03:55:46,074 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0337be0f939be2a2b593eefecee32758 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-29 03:55:46,074 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0337be0f939be2a2b593eefecee32758, store=A 2023-03-29 03:55:46,074 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:55:46,074 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0337be0f939be2a2b593eefecee32758, store=B 2023-03-29 03:55:46,074 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:55:46,075 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0337be0f939be2a2b593eefecee32758, store=C 2023-03-29 03:55:46,075 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:55:46,081 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=33 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/A/1e00e4ccd2934f819164fee5c6951415 2023-03-29 03:55:46,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on 0337be0f939be2a2b593eefecee32758 2023-03-29 03:55:46,289 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. as already flushing 2023-03-29 03:55:46,300 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:46,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42024 deadline: 1680062206294, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:46,303 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:46,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42010 deadline: 1680062206298, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:46,304 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:46,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41960 deadline: 1680062206299, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:46,305 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:46,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42004 deadline: 1680062206300, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:46,305 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:46,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41942 deadline: 1680062206301, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:46,402 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:46,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42024 deadline: 1680062206402, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:46,407 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:46,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42010 deadline: 1680062206405, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:46,407 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:46,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41960 deadline: 1680062206405, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:46,407 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:46,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41942 deadline: 1680062206406, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:46,408 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:46,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42004 deadline: 1680062206406, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:46,494 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=33 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/B/2614cf7ffcc743e3bdd7fbc3310a4690 2023-03-29 03:55:46,607 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:46,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42024 deadline: 1680062206605, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:46,613 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:46,613 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:46,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42010 deadline: 1680062206608, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:46,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42004 deadline: 1680062206608, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:46,614 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:46,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41960 deadline: 1680062206609, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:46,614 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:46,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41942 deadline: 1680062206609, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:46,902 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=33 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/C/ec12ce726f5e4f96a8d5920ca5912b90 2023-03-29 03:55:46,906 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/A/1e00e4ccd2934f819164fee5c6951415 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/1e00e4ccd2934f819164fee5c6951415 2023-03-29 03:55:46,909 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/1e00e4ccd2934f819164fee5c6951415, entries=150, sequenceid=33, filesize=11.6 K 2023-03-29 03:55:46,910 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/B/2614cf7ffcc743e3bdd7fbc3310a4690 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/2614cf7ffcc743e3bdd7fbc3310a4690 2023-03-29 03:55:46,912 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:46,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42024 deadline: 1680062206910, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:46,913 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/2614cf7ffcc743e3bdd7fbc3310a4690, entries=150, sequenceid=33, filesize=11.6 K 2023-03-29 03:55:46,914 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/C/ec12ce726f5e4f96a8d5920ca5912b90 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/ec12ce726f5e4f96a8d5920ca5912b90 2023-03-29 03:55:46,917 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:46,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41960 deadline: 1680062206915, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:46,917 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:46,917 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/ec12ce726f5e4f96a8d5920ca5912b90, entries=150, sequenceid=33, filesize=11.6 K 2023-03-29 03:55:46,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41942 deadline: 1680062206915, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:46,918 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 0337be0f939be2a2b593eefecee32758 in 844ms, sequenceid=33, compaction requested=false 2023-03-29 03:55:46,918 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0337be0f939be2a2b593eefecee32758: 2023-03-29 03:55:46,918 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:46,919 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:46,919 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:46,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on 0337be0f939be2a2b593eefecee32758 2023-03-29 03:55:46,919 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:46,919 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 0337be0f939be2a2b593eefecee32758 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-03-29 03:55:46,919 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:46,919 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:46,919 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0337be0f939be2a2b593eefecee32758, store=A 2023-03-29 03:55:46,919 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:55:46,919 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0337be0f939be2a2b593eefecee32758, store=B 2023-03-29 03:55:46,919 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:55:46,919 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0337be0f939be2a2b593eefecee32758, store=C 2023-03-29 03:55:46,919 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:55:46,920 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:46,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:46,921 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:46,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:46,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:46,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:46,921 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:46,921 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:46,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:46,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:46,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:46,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:46,923 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:46,923 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@24655f53[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:46,923 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:46,923 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:46,925 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:46,925 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:46,926 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:46,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:46,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:46,926 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:46,926 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:46,926 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:46,929 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:46,929 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:46,929 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:46,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:46,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:46,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:46,929 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:46,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:46,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:46,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:46,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:46,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:46,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:46,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:46,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:46,932 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:46,932 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:46,932 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:46,932 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:46,932 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:46,932 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:46,933 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:46,933 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:46,933 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:46,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:46,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:46,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:46,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:46,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:46,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:46,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:46,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:46,934 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:46,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:46,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:46,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:46,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:46,935 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:46,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:46,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:46,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:46,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:46,939 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:46,939 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:46,939 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:46,939 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:46,939 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:46,939 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:46,939 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:46,939 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:46,939 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:46,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:46,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:46,939 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:46,939 DEBUG [Thread-3261] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:46,939 DEBUG [Thread-3261] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:46,939 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:55:46,939 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:46,940 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:46,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:46,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:46,979 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:46,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42004 deadline: 1680062206972, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:46,979 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:46,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42010 deadline: 1680062206976, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:47,040 DEBUG [Thread-3261] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:47,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:47,042 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:47,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:47,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:47,043 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:47,043 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:47,043 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:47,043 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:47,043 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:47,043 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:47,045 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:47,045 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:47,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:47,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:47,045 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:47,045 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:47,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:47,046 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:47,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:47,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:47,046 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:47,046 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:47,046 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:47,046 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:47,047 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:47,047 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:47,047 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:47,047 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:47,047 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:47,047 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. started... 2023-03-29 03:55:47,047 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. as already flushing 2023-03-29 03:55:47,047 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:47,047 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:47,047 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:47,047 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:47,047 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:47,047 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:47,050 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:47,050 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:47,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:47,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:47,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:47,050 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:47,050 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:47,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:47,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:47,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:47,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:47,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:47,052 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:47,052 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70c473dc[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:47,052 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:47,052 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:47,053 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:47,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:47,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:47,053 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:47,053 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:47,053 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:47,053 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:47,053 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:47,054 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:47,054 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:47,054 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:47,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:47,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:47,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:47,054 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:47,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:47,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:47,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:47,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:47,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:47,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:47,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:47,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:47,057 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:47,057 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:47,057 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:47,057 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:47,057 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:47,057 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:47,058 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:47,058 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:47,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:47,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:47,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:47,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:47,058 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:47,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:47,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:47,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:47,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:47,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:47,059 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:47,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:47,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:47,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:47,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:47,060 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:47,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:47,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:47,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:47,063 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:47,063 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:47,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:47,063 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:47,063 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:47,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:47,063 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:47,064 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:47,063 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:47,064 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:47,064 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:47,064 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:47,064 DEBUG [Thread-3261] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:47,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:47,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:47,064 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:47,064 DEBUG [Thread-3261] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:47,064 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:47,084 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:47,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42004 deadline: 1680062207080, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:47,085 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:47,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42010 deadline: 1680062207080, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:47,164 DEBUG [Thread-3261] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:47,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:47,166 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:47,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:47,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:47,167 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:47,167 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:47,167 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:47,167 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:47,168 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:47,168 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:47,169 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:47,169 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:47,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:47,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:47,170 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:47,170 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:47,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:47,170 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:47,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:47,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:47,170 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:47,171 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:47,171 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:47,171 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:47,171 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:47,171 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:47,171 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:47,171 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:47,171 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:47,171 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. started... 2023-03-29 03:55:47,171 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. as already flushing 2023-03-29 03:55:47,171 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:47,171 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:47,171 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:47,171 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:47,171 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:47,171 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:47,176 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:47,176 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:47,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:47,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:47,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:47,176 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:47,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:47,176 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:47,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:47,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:47,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:47,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:47,177 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:47,177 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5716bb3c[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:47,177 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:47,177 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:47,180 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:47,180 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:47,180 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:47,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:47,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:47,180 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:47,180 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:47,180 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:47,182 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:47,182 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:47,182 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:47,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:47,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:47,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:47,182 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:47,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:47,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:47,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:47,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:47,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:47,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:47,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:47,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:47,184 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:47,184 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:47,184 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:47,184 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:47,184 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:47,184 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:47,185 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:47,185 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:47,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:47,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:47,185 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:47,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:47,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:47,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:47,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:47,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:47,186 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:47,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:47,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:47,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:47,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:47,187 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:47,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:47,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:47,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:47,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:47,189 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:47,189 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:47,189 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:47,189 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:47,189 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:47,189 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:47,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:47,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:47,189 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:47,189 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:55:47,189 DEBUG [Thread-3261] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:47,189 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:47,190 DEBUG [Thread-3261] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:47,190 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:47,190 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:47,190 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:47,190 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:47,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:47,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:47,290 DEBUG [Thread-3261] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:47,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:47,292 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:47,292 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:47,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42004 deadline: 1680062207285, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:47,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:47,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:47,293 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:47,293 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:47,292 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:47,293 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:47,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42010 deadline: 1680062207286, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:47,293 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:47,293 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:47,293 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:47,295 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:47,295 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:47,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:47,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:47,296 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:47,296 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:47,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:47,296 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:47,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:47,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:47,296 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:47,297 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:47,297 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:47,297 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:47,297 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:47,297 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:47,297 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:47,297 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:47,297 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:47,297 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. started... 2023-03-29 03:55:47,297 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. as already flushing 2023-03-29 03:55:47,297 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:47,297 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:47,297 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:47,297 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:47,297 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:47,297 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:47,299 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:47,299 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:47,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:47,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:47,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:47,299 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:47,299 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:47,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:47,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:47,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:47,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:47,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:47,300 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:47,301 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@610e09e1[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:47,301 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:47,301 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:47,302 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:47,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:47,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:47,302 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:47,302 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:47,302 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:47,302 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:47,302 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:47,303 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:47,303 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:47,303 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:47,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:47,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:47,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:47,303 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:47,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:47,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:47,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:47,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:47,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:47,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:47,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:47,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:47,305 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:47,305 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:47,306 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:47,306 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:47,306 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:47,306 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:47,308 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:47,308 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:47,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:47,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:47,308 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:47,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:47,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:47,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:47,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:47,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:47,309 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:47,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:47,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:47,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:47,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:47,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:47,311 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:47,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:47,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:47,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:47,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:47,314 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:47,314 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:47,314 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:47,314 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:55:47,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:47,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:47,314 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:47,314 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:47,314 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:47,314 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:47,314 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:47,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:47,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:47,314 DEBUG [Thread-3261] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:47,314 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:47,314 DEBUG [Thread-3261] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:47,314 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:47,315 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:47,315 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:47,346 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=47 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/A/870bab9e58f14816ac5fa8dd8f3319c3 2023-03-29 03:55:47,354 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=47 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/B/7ba395cadaaa42cfb3593a08fb446132 2023-03-29 03:55:47,415 DEBUG [Thread-3261] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:47,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:47,427 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:47,430 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:47,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42024 deadline: 1680062207413, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:47,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:47,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:47,431 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:47,431 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:47,431 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:47,431 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:47,431 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:47,431 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:47,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41942 deadline: 1680062207426, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:47,432 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:47,433 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:47,433 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:47,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:47,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:47,433 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:47,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:47,433 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:47,434 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:47,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:47,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:47,434 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:47,434 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:47,434 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:47,434 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:47,434 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:47,434 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:47,434 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:47,434 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:47,434 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. started... 2023-03-29 03:55:47,435 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. as already flushing 2023-03-29 03:55:47,435 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:47,435 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:47,435 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:47,435 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:47,435 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:47,435 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:47,436 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:47,436 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:47,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:47,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:47,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:47,437 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:47,437 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:47,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:47,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:47,437 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:47,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41960 deadline: 1680062207431, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:47,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:47,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:47,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:47,438 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:47,438 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f1f4d54[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:47,438 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:47,438 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:47,440 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:47,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:47,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:47,441 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:47,441 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:47,441 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:47,441 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:47,441 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:47,442 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:47,442 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:47,442 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:47,442 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:47,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:47,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:47,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:47,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:47,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:47,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:47,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:47,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:47,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:47,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:47,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:47,444 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:47,444 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:47,444 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:47,444 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:47,444 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:47,445 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:47,446 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:47,446 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:47,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:47,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:47,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:47,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:47,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:47,446 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:47,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:47,447 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:47,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:47,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:47,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:47,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:47,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:47,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:47,448 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:47,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:47,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:47,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:47,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:47,450 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:47,450 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:47,450 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:47,450 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:47,450 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:47,450 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:47,450 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:47,450 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:47,450 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:47,450 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:47,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:47,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:47,450 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:47,450 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:47,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:47,450 DEBUG [Thread-3261] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:47,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:47,450 DEBUG [Thread-3261] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:47,473 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-03-29 03:55:47,551 DEBUG [Thread-3261] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:47,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:47,553 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:47,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:47,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:47,554 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:47,554 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:47,554 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:47,554 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:47,555 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:47,555 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:47,557 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:47,557 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:47,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:47,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:47,558 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:47,558 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:47,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:47,558 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:47,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:47,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:47,558 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:47,558 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:47,558 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:47,559 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:47,559 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:47,559 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:47,559 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:47,559 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:47,559 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:47,559 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. started... 2023-03-29 03:55:47,559 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. as already flushing 2023-03-29 03:55:47,559 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:47,559 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:47,559 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:47,559 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:47,559 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:47,559 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:47,561 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:47,561 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:47,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:47,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:47,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:47,561 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:47,561 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:47,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:47,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:47,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:47,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:47,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:47,562 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:47,563 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3c1689f8[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:47,563 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:47,563 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:47,564 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:47,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:47,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:47,564 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:47,564 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:47,564 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:47,564 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:47,564 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:47,570 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:47,570 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:47,570 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:47,570 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:47,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:47,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:47,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:47,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:47,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:47,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:47,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:47,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:47,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:47,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:47,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:47,573 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:47,573 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:47,573 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:47,573 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:47,573 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:47,573 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:47,574 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:47,575 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:47,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:47,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:47,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:47,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:47,575 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:47,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:47,575 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:47,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:47,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:47,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:47,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:47,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:47,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:47,576 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:47,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:47,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:47,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:47,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:47,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:47,578 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:47,578 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:47,578 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:47,578 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:47,578 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:47,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:47,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:47,579 DEBUG [Thread-3261] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:47,579 DEBUG [Thread-3261] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:47,578 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:47,578 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:47,578 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:47,579 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:47,579 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:47,579 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:47,579 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:47,579 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:55:47,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:47,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:47,596 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:47,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42004 deadline: 1680062207593, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:47,599 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:47,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42010 deadline: 1680062207596, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:47,679 DEBUG [Thread-3261] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:47,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:47,681 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:47,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:47,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:47,682 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:47,682 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:47,682 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:47,682 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:47,683 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:47,683 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:47,684 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:47,684 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:47,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:47,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:47,684 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:47,684 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:47,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:47,685 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:47,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:47,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:47,685 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:47,685 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:47,685 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:47,685 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:47,686 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:47,686 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:47,686 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:47,686 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:47,686 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:47,686 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. started... 2023-03-29 03:55:47,686 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. as already flushing 2023-03-29 03:55:47,686 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:47,687 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:47,687 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:47,687 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:47,687 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:47,687 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:47,689 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:47,690 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:47,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:47,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:47,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:47,690 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:47,690 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:47,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:47,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:47,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:47,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:47,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:47,691 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:47,692 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e3b1a1d[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:47,692 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:47,692 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:47,693 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:47,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:47,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:47,693 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:47,693 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:47,693 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:47,693 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:47,693 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:47,694 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:47,694 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:47,694 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:47,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:47,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:47,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:47,694 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:47,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:47,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:47,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:47,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:47,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:47,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:47,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:47,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:47,698 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:47,698 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:47,698 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:47,698 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:47,698 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:47,698 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:47,700 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:47,700 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:47,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:47,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:47,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:47,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:47,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:47,700 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:47,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:47,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:47,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:47,701 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:47,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:47,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:47,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:47,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:47,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:47,702 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:47,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:47,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:47,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:47,706 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:47,706 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:47,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:47,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:47,706 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:47,706 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:47,706 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:47,706 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:47,706 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:47,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:47,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:47,706 DEBUG [Thread-3261] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:47,706 DEBUG [Thread-3261] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:47,706 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:47,707 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:47,707 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:47,707 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:47,707 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:47,762 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=47 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/C/3a317a92178b45a99a492e78a5ff7e2b 2023-03-29 03:55:47,766 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/A/870bab9e58f14816ac5fa8dd8f3319c3 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/870bab9e58f14816ac5fa8dd8f3319c3 2023-03-29 03:55:47,768 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/870bab9e58f14816ac5fa8dd8f3319c3, entries=200, sequenceid=47, filesize=13.9 K 2023-03-29 03:55:47,769 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/B/7ba395cadaaa42cfb3593a08fb446132 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/7ba395cadaaa42cfb3593a08fb446132 2023-03-29 03:55:47,772 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/7ba395cadaaa42cfb3593a08fb446132, entries=150, sequenceid=47, filesize=11.6 K 2023-03-29 03:55:47,773 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/C/3a317a92178b45a99a492e78a5ff7e2b as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/3a317a92178b45a99a492e78a5ff7e2b 2023-03-29 03:55:47,776 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/3a317a92178b45a99a492e78a5ff7e2b, entries=150, sequenceid=47, filesize=11.6 K 2023-03-29 03:55:47,776 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 0337be0f939be2a2b593eefecee32758 in 857ms, sequenceid=47, compaction requested=true 2023-03-29 03:55:47,776 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 0337be0f939be2a2b593eefecee32758: 2023-03-29 03:55:47,777 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:55:47,777 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-29 03:55:47,777 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-29 03:55:47,777 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:55:47,777 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:55:47,777 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37998 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-29 03:55:47,777 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1912): 0337be0f939be2a2b593eefecee32758/A is initiating minor compaction (all files) 2023-03-29 03:55:47,777 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-29 03:55:47,778 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 0337be0f939be2a2b593eefecee32758/A in TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:47,778 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1912): 0337be0f939be2a2b593eefecee32758/B is initiating minor compaction (all files) 2023-03-29 03:55:47,778 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/864bc06b60e74a85addefd04b73687ac, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/1e00e4ccd2934f819164fee5c6951415, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/870bab9e58f14816ac5fa8dd8f3319c3] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp, totalSize=37.1 K 2023-03-29 03:55:47,778 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0337be0f939be2a2b593eefecee32758/B in TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:47,778 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/2cda48bbe6884a9ebddda4f078f5687d, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/2614cf7ffcc743e3bdd7fbc3310a4690, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/7ba395cadaaa42cfb3593a08fb446132] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp, totalSize=34.8 K 2023-03-29 03:55:47,778 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 864bc06b60e74a85addefd04b73687ac, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=14, earliestPutTs=1680062145152 2023-03-29 03:55:47,778 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 2cda48bbe6884a9ebddda4f078f5687d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=14, earliestPutTs=1680062145152 2023-03-29 03:55:47,778 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 1e00e4ccd2934f819164fee5c6951415, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=33, earliestPutTs=1680062145160 2023-03-29 03:55:47,778 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 2614cf7ffcc743e3bdd7fbc3310a4690, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=33, earliestPutTs=1680062145160 2023-03-29 03:55:47,779 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 870bab9e58f14816ac5fa8dd8f3319c3, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=47, earliestPutTs=1680062146293 2023-03-29 03:55:47,779 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 7ba395cadaaa42cfb3593a08fb446132, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=47, earliestPutTs=1680062146293 2023-03-29 03:55:47,793 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] throttle.PressureAwareThroughputController(145): 0337be0f939be2a2b593eefecee32758#B#compaction#436 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:55:47,802 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0337be0f939be2a2b593eefecee32758#A#compaction#437 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:55:47,807 DEBUG [Thread-3261] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:47,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:47,810 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:47,811 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/B/810d71e247684531a42f751bff5c4531 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/810d71e247684531a42f751bff5c4531 2023-03-29 03:55:47,816 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0337be0f939be2a2b593eefecee32758/B of 0337be0f939be2a2b593eefecee32758 into 810d71e247684531a42f751bff5c4531(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:55:47,816 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0337be0f939be2a2b593eefecee32758: 2023-03-29 03:55:47,816 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758., storeName=0337be0f939be2a2b593eefecee32758/B, priority=13, startTime=1680062147777; duration=0sec 2023-03-29 03:55:47,816 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:55:47,816 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-29 03:55:47,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:47,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:47,817 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:47,818 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:47,818 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:47,818 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:47,818 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-29 03:55:47,818 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1912): 0337be0f939be2a2b593eefecee32758/C is initiating minor compaction (all files) 2023-03-29 03:55:47,818 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:47,818 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0337be0f939be2a2b593eefecee32758/C in TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:47,818 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:47,818 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/ff07ce7ad8024f97a6be2f47b92a3b03, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/ec12ce726f5e4f96a8d5920ca5912b90, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/3a317a92178b45a99a492e78a5ff7e2b] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp, totalSize=34.8 K 2023-03-29 03:55:47,818 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting ff07ce7ad8024f97a6be2f47b92a3b03, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=14, earliestPutTs=1680062145152 2023-03-29 03:55:47,819 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting ec12ce726f5e4f96a8d5920ca5912b90, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=33, earliestPutTs=1680062145160 2023-03-29 03:55:47,819 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 3a317a92178b45a99a492e78a5ff7e2b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=47, earliestPutTs=1680062146293 2023-03-29 03:55:47,819 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:47,819 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:47,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:47,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:47,820 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:47,820 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:47,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:47,820 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:47,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:47,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:47,821 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:47,832 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:47,832 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:47,832 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:47,832 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:47,832 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:47,833 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:47,833 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:47,836 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/A/5cf050c8a8df41719a04e46bc3dcc78e as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/5cf050c8a8df41719a04e46bc3dcc78e 2023-03-29 03:55:47,836 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:47,836 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. started... 2023-03-29 03:55:47,837 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] throttle.PressureAwareThroughputController(145): 0337be0f939be2a2b593eefecee32758#C#compaction#438 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:55:47,838 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0337be0f939be2a2b593eefecee32758 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-03-29 03:55:47,839 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0337be0f939be2a2b593eefecee32758, store=A 2023-03-29 03:55:47,839 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:55:47,839 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0337be0f939be2a2b593eefecee32758, store=B 2023-03-29 03:55:47,839 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:55:47,839 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0337be0f939be2a2b593eefecee32758, store=C 2023-03-29 03:55:47,839 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:55:47,845 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0337be0f939be2a2b593eefecee32758/A of 0337be0f939be2a2b593eefecee32758 into 5cf050c8a8df41719a04e46bc3dcc78e(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:55:47,845 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0337be0f939be2a2b593eefecee32758: 2023-03-29 03:55:47,846 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758., storeName=0337be0f939be2a2b593eefecee32758/A, priority=13, startTime=1680062147777; duration=0sec 2023-03-29 03:55:47,846 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:55:48,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on 0337be0f939be2a2b593eefecee32758 2023-03-29 03:55:48,105 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. as already flushing 2023-03-29 03:55:48,148 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:48,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42004 deadline: 1680062208144, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:48,148 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:48,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42010 deadline: 1680062208144, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:48,251 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:48,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42004 deadline: 1680062208249, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:48,251 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:48,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42010 deadline: 1680062208249, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:48,257 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/C/61f489be14cd44eda718675e5f141996 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/61f489be14cd44eda718675e5f141996 2023-03-29 03:55:48,261 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0337be0f939be2a2b593eefecee32758/C of 0337be0f939be2a2b593eefecee32758 into 61f489be14cd44eda718675e5f141996(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:55:48,261 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0337be0f939be2a2b593eefecee32758: 2023-03-29 03:55:48,261 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758., storeName=0337be0f939be2a2b593eefecee32758/C, priority=13, startTime=1680062147777; duration=0sec 2023-03-29 03:55:48,261 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:55:48,290 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=66 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/A/6e49ac1f32944776aad64cae500aa7b5 2023-03-29 03:55:48,432 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:48,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42024 deadline: 1680062208432, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:48,441 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:48,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41942 deadline: 1680062208440, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:48,441 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:48,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41960 deadline: 1680062208440, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:48,454 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:48,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42010 deadline: 1680062208452, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:48,454 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:48,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42004 deadline: 1680062208453, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:48,700 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=66 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/B/4da76b6e515a45128573ff35877ed44b 2023-03-29 03:55:48,710 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=66 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/C/2847721ff5964bb18fb5aba244c7f802 2023-03-29 03:55:48,713 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/A/6e49ac1f32944776aad64cae500aa7b5 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/6e49ac1f32944776aad64cae500aa7b5 2023-03-29 03:55:48,717 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/6e49ac1f32944776aad64cae500aa7b5, entries=150, sequenceid=66, filesize=11.6 K 2023-03-29 03:55:48,717 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/B/4da76b6e515a45128573ff35877ed44b as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/4da76b6e515a45128573ff35877ed44b 2023-03-29 03:55:48,720 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/4da76b6e515a45128573ff35877ed44b, entries=150, sequenceid=66, filesize=11.6 K 2023-03-29 03:55:48,722 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/C/2847721ff5964bb18fb5aba244c7f802 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/2847721ff5964bb18fb5aba244c7f802 2023-03-29 03:55:48,724 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/2847721ff5964bb18fb5aba244c7f802, entries=150, sequenceid=66, filesize=11.6 K 2023-03-29 03:55:48,725 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 0337be0f939be2a2b593eefecee32758 in 887ms, sequenceid=66, compaction requested=false 2023-03-29 03:55:48,725 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0337be0f939be2a2b593eefecee32758: 2023-03-29 03:55:48,725 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:48,725 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:48,725 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:48,725 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:48,725 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:48,726 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:48,727 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:48,727 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:48,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:48,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:48,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:48,728 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:48,728 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:48,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:48,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:48,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:48,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:48,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:48,729 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:48,729 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4feb413a[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:48,729 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:48,730 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:48,731 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:48,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:48,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:48,731 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:48,731 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:48,731 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:48,732 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:48,732 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:48,733 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:48,733 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:48,733 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:48,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:48,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:48,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:48,733 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:48,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:48,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:48,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:48,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:48,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:48,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:48,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:48,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:48,736 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:48,736 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:48,736 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:48,736 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:48,736 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:48,736 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:48,738 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:48,738 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:48,738 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:48,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:48,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:48,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:48,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:48,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:48,738 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:48,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:48,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:48,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:48,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:48,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:48,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:48,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:48,740 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:48,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:48,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:48,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:48,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:48,743 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:48,743 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:48,743 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:48,743 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:48,743 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:48,743 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:48,743 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:48,743 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:48,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:48,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:48,743 DEBUG [Thread-3261] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:48,743 DEBUG [Thread-3261] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:48,743 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:48,744 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:48,744 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:48,744 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:48,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:48,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:48,744 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:55:48,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on 0337be0f939be2a2b593eefecee32758 2023-03-29 03:55:48,759 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 0337be0f939be2a2b593eefecee32758 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-03-29 03:55:48,759 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0337be0f939be2a2b593eefecee32758, store=A 2023-03-29 03:55:48,759 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:55:48,759 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0337be0f939be2a2b593eefecee32758, store=B 2023-03-29 03:55:48,759 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:55:48,759 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0337be0f939be2a2b593eefecee32758, store=C 2023-03-29 03:55:48,759 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:55:48,766 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=83 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/A/f7621e1159ef42bb8ebb6ff0f78b57d3 2023-03-29 03:55:48,774 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=83 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/B/158c8b947a6245478838e4719cbe8341 2023-03-29 03:55:48,801 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:48,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42004 deadline: 1680062208797, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:48,803 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:48,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42010 deadline: 1680062208798, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:48,844 DEBUG [Thread-3261] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:48,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:48,846 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:48,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:48,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:48,847 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:48,847 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:48,847 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:48,847 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:48,847 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:48,847 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:48,849 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:48,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:48,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:48,849 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:48,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:48,849 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:48,849 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:48,849 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:48,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:48,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:48,850 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:48,850 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:48,850 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:48,850 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:48,850 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:48,850 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:48,850 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:48,850 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:48,850 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:48,850 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. started... 2023-03-29 03:55:48,851 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. as already flushing 2023-03-29 03:55:48,851 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:48,851 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:48,851 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:48,851 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:48,851 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:48,851 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:48,858 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:48,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:48,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:48,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:48,858 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:48,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:48,859 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:48,859 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:48,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:48,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:48,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:48,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:48,860 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:48,860 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a0bd07e[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:48,860 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:48,860 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:48,861 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:48,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:48,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:48,862 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:48,862 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:48,862 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:48,862 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:48,862 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:48,863 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:48,863 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:48,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:48,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:48,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:48,863 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:48,863 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:48,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:48,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:48,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:48,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:48,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:48,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:48,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:48,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:48,866 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:48,866 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:48,866 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:48,866 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:48,866 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:48,866 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:48,877 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:48,877 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:48,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:48,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:48,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:48,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:48,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:48,877 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:48,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:48,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:48,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:48,878 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:48,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:48,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:48,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:48,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:48,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:48,879 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:48,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:48,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:48,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:48,882 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:48,883 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:48,883 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:48,883 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:48,883 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:48,883 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:55:48,883 DEBUG [Thread-3261] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:48,883 DEBUG [Thread-3261] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:48,883 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:48,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:48,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:48,883 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:48,883 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:48,883 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:48,883 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:48,883 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:48,883 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:48,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:48,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:48,907 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:48,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42004 deadline: 1680062208902, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:48,907 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:48,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42010 deadline: 1680062208904, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:48,983 DEBUG [Thread-3261] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:48,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:48,985 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:48,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:48,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:48,986 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:48,986 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:48,986 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:48,986 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:48,987 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:48,987 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:48,988 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:48,988 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:48,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:48,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:48,988 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:48,988 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:48,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:48,989 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:48,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:48,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:48,989 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:48,989 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:48,989 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:48,989 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:48,989 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:48,989 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:48,990 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:48,990 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:48,990 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:48,990 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. started... 2023-03-29 03:55:48,990 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. as already flushing 2023-03-29 03:55:48,990 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:48,990 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:48,990 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:48,990 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:48,990 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:48,990 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:48,991 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:48,991 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:48,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:48,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:48,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:48,992 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:48,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:48,992 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:48,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:48,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:48,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:48,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:48,993 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:48,993 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@52dadb8e[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:48,993 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:48,993 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:48,995 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:48,995 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:48,996 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:48,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:48,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:48,996 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:48,996 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:48,996 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:48,998 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:48,998 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:48,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:48,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:48,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:48,998 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:48,998 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:48,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:48,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:48,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:48,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:48,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:49,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:49,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:49,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:49,000 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:49,000 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:49,000 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:49,000 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:49,000 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:49,000 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:49,002 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:49,002 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:49,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:49,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:49,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:49,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:49,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:49,002 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:49,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:49,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:49,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:49,003 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:49,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:49,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:49,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:49,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:49,004 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:49,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:49,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:49,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:49,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:49,006 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:49,006 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:55:49,006 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:49,006 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:49,006 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:49,006 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:49,006 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:49,006 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:49,007 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:49,007 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:49,007 DEBUG [Thread-3261] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:49,007 DEBUG [Thread-3261] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:49,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:49,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:49,007 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:49,007 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:49,007 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:49,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:49,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:49,107 DEBUG [Thread-3261] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:49,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:49,109 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:49,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:49,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:49,110 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:49,110 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:49,110 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:49,110 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:49,110 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:49,110 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:49,111 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:49,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42004 deadline: 1680062209108, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:49,111 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:49,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42010 deadline: 1680062209108, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:49,113 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:49,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:49,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:49,113 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:49,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:49,113 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:49,113 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:49,113 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:49,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:49,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:49,114 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:49,114 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:49,114 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:49,114 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:49,114 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:49,114 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:49,114 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:49,114 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:49,114 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:49,114 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. started... 2023-03-29 03:55:49,114 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. as already flushing 2023-03-29 03:55:49,115 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:49,115 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:49,115 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:49,115 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:49,115 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:49,115 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:49,116 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:49,116 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:49,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:49,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:49,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:49,116 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:49,117 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:49,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:49,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:49,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:49,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:49,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:49,118 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:49,118 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5437c83a[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:49,118 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:49,118 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:49,119 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:49,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:49,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:49,119 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:49,119 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:49,119 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:49,119 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:49,119 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:49,121 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:49,121 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:49,121 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:49,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:49,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:49,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:49,121 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:49,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:49,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:49,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:49,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:49,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:49,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:49,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:49,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:49,123 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:49,123 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:49,123 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:49,123 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:49,123 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:49,123 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:49,126 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:49,126 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:49,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:49,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:49,126 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:49,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:49,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:49,126 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:49,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:49,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:49,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:49,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:49,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:49,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:49,128 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:49,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:49,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:49,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:49,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:49,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:49,130 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:49,130 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:49,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:49,130 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:49,130 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:49,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:49,130 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:49,130 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:49,131 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:55:49,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:49,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:49,131 DEBUG [Thread-3261] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:49,131 DEBUG [Thread-3261] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:49,131 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:49,131 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:49,131 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:49,131 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:49,131 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:49,132 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:49,183 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=83 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/C/7854b5d0a7bd4202ba02a1fc8b3dd62a 2023-03-29 03:55:49,187 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/A/f7621e1159ef42bb8ebb6ff0f78b57d3 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/f7621e1159ef42bb8ebb6ff0f78b57d3 2023-03-29 03:55:49,190 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/f7621e1159ef42bb8ebb6ff0f78b57d3, entries=200, sequenceid=83, filesize=13.9 K 2023-03-29 03:55:49,191 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/B/158c8b947a6245478838e4719cbe8341 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/158c8b947a6245478838e4719cbe8341 2023-03-29 03:55:49,194 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/158c8b947a6245478838e4719cbe8341, entries=150, sequenceid=83, filesize=11.6 K 2023-03-29 03:55:49,195 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/C/7854b5d0a7bd4202ba02a1fc8b3dd62a as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/7854b5d0a7bd4202ba02a1fc8b3dd62a 2023-03-29 03:55:49,199 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/7854b5d0a7bd4202ba02a1fc8b3dd62a, entries=150, sequenceid=83, filesize=11.6 K 2023-03-29 03:55:49,199 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 0337be0f939be2a2b593eefecee32758 in 441ms, sequenceid=83, compaction requested=true 2023-03-29 03:55:49,199 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 0337be0f939be2a2b593eefecee32758: 2023-03-29 03:55:49,200 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:55:49,200 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-29 03:55:49,200 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-29 03:55:49,200 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:55:49,200 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:55:49,201 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 38101 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-29 03:55:49,201 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35761 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-29 03:55:49,201 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1912): 0337be0f939be2a2b593eefecee32758/A is initiating minor compaction (all files) 2023-03-29 03:55:49,201 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1912): 0337be0f939be2a2b593eefecee32758/B is initiating minor compaction (all files) 2023-03-29 03:55:49,201 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 0337be0f939be2a2b593eefecee32758/A in TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:49,201 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0337be0f939be2a2b593eefecee32758/B in TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:49,201 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/5cf050c8a8df41719a04e46bc3dcc78e, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/6e49ac1f32944776aad64cae500aa7b5, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/f7621e1159ef42bb8ebb6ff0f78b57d3] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp, totalSize=37.2 K 2023-03-29 03:55:49,201 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/810d71e247684531a42f751bff5c4531, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/4da76b6e515a45128573ff35877ed44b, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/158c8b947a6245478838e4719cbe8341] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp, totalSize=34.9 K 2023-03-29 03:55:49,201 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 5cf050c8a8df41719a04e46bc3dcc78e, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=47, earliestPutTs=1680062146293 2023-03-29 03:55:49,201 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 810d71e247684531a42f751bff5c4531, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=47, earliestPutTs=1680062146293 2023-03-29 03:55:49,202 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 6e49ac1f32944776aad64cae500aa7b5, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=66, earliestPutTs=1680062146951 2023-03-29 03:55:49,202 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 4da76b6e515a45128573ff35877ed44b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=66, earliestPutTs=1680062146951 2023-03-29 03:55:49,202 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting f7621e1159ef42bb8ebb6ff0f78b57d3, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1680062148109 2023-03-29 03:55:49,202 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 158c8b947a6245478838e4719cbe8341, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1680062148123 2023-03-29 03:55:49,209 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] throttle.PressureAwareThroughputController(145): 0337be0f939be2a2b593eefecee32758#B#compaction#445 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:55:49,211 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0337be0f939be2a2b593eefecee32758#A#compaction#446 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:55:49,231 DEBUG [Thread-3261] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:49,232 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/B/ff38438e894640a9a40125a321c52993 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/ff38438e894640a9a40125a321c52993 2023-03-29 03:55:49,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:49,234 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:49,234 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/A/1416602e3e27463ab58f12fe33aeed04 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/1416602e3e27463ab58f12fe33aeed04 2023-03-29 03:55:49,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:49,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:49,235 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:49,236 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:49,236 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:49,236 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:49,236 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:49,237 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:49,238 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0337be0f939be2a2b593eefecee32758/B of 0337be0f939be2a2b593eefecee32758 into ff38438e894640a9a40125a321c52993(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:55:49,238 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:49,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:49,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:49,238 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:49,238 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0337be0f939be2a2b593eefecee32758: 2023-03-29 03:55:49,239 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758., storeName=0337be0f939be2a2b593eefecee32758/B, priority=13, startTime=1680062149200; duration=0sec 2023-03-29 03:55:49,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:49,239 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:55:49,239 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-29 03:55:49,239 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:49,239 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:49,240 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:49,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:49,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:49,240 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:49,240 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0337be0f939be2a2b593eefecee32758/A of 0337be0f939be2a2b593eefecee32758 into 1416602e3e27463ab58f12fe33aeed04(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:55:49,240 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0337be0f939be2a2b593eefecee32758: 2023-03-29 03:55:49,240 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758., storeName=0337be0f939be2a2b593eefecee32758/A, priority=13, startTime=1680062149199; duration=0sec 2023-03-29 03:55:49,240 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:55:49,240 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:49,241 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:49,241 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:49,241 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:49,241 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:49,241 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35761 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-29 03:55:49,241 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1912): 0337be0f939be2a2b593eefecee32758/C is initiating minor compaction (all files) 2023-03-29 03:55:49,241 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0337be0f939be2a2b593eefecee32758/C in TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:49,241 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/61f489be14cd44eda718675e5f141996, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/2847721ff5964bb18fb5aba244c7f802, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/7854b5d0a7bd4202ba02a1fc8b3dd62a] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp, totalSize=34.9 K 2023-03-29 03:55:49,241 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 61f489be14cd44eda718675e5f141996, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=47, earliestPutTs=1680062146293 2023-03-29 03:55:49,242 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 2847721ff5964bb18fb5aba244c7f802, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=66, earliestPutTs=1680062146951 2023-03-29 03:55:49,242 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 7854b5d0a7bd4202ba02a1fc8b3dd62a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1680062148123 2023-03-29 03:55:49,244 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:49,244 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:49,244 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:49,244 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. started... 2023-03-29 03:55:49,244 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0337be0f939be2a2b593eefecee32758 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-03-29 03:55:49,244 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0337be0f939be2a2b593eefecee32758, store=A 2023-03-29 03:55:49,244 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:55:49,245 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0337be0f939be2a2b593eefecee32758, store=B 2023-03-29 03:55:49,245 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:55:49,245 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0337be0f939be2a2b593eefecee32758, store=C 2023-03-29 03:55:49,245 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:55:49,247 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] throttle.PressureAwareThroughputController(145): 0337be0f939be2a2b593eefecee32758#C#compaction#447 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:55:49,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on 0337be0f939be2a2b593eefecee32758 2023-03-29 03:55:49,416 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. as already flushing 2023-03-29 03:55:49,443 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:49,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42004 deadline: 1680062209438, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:49,444 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:49,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42010 deadline: 1680062209443, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:49,546 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:49,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42004 deadline: 1680062209544, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:49,549 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:49,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42010 deadline: 1680062209545, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:49,658 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=102 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/A/400588da025049dbaaeb6f7e5d267f09 2023-03-29 03:55:49,663 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/C/9a96ecf7a8134fce99f2d5c33f57647f as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/9a96ecf7a8134fce99f2d5c33f57647f 2023-03-29 03:55:49,667 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0337be0f939be2a2b593eefecee32758/C of 0337be0f939be2a2b593eefecee32758 into 9a96ecf7a8134fce99f2d5c33f57647f(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:55:49,667 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0337be0f939be2a2b593eefecee32758: 2023-03-29 03:55:49,667 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758., storeName=0337be0f939be2a2b593eefecee32758/C, priority=13, startTime=1680062149200; duration=0sec 2023-03-29 03:55:49,667 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:55:49,669 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=102 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/B/f77132a4406e418bb8ea6dc33ec073d1 2023-03-29 03:55:49,678 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=102 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/C/c29e41d57b164baa98de1a169f6a6597 2023-03-29 03:55:49,681 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/A/400588da025049dbaaeb6f7e5d267f09 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/400588da025049dbaaeb6f7e5d267f09 2023-03-29 03:55:49,684 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/400588da025049dbaaeb6f7e5d267f09, entries=150, sequenceid=102, filesize=11.6 K 2023-03-29 03:55:49,685 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/B/f77132a4406e418bb8ea6dc33ec073d1 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/f77132a4406e418bb8ea6dc33ec073d1 2023-03-29 03:55:49,688 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/f77132a4406e418bb8ea6dc33ec073d1, entries=150, sequenceid=102, filesize=11.6 K 2023-03-29 03:55:49,689 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/C/c29e41d57b164baa98de1a169f6a6597 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/c29e41d57b164baa98de1a169f6a6597 2023-03-29 03:55:49,691 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/c29e41d57b164baa98de1a169f6a6597, entries=150, sequenceid=102, filesize=11.6 K 2023-03-29 03:55:49,692 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 0337be0f939be2a2b593eefecee32758 in 448ms, sequenceid=102, compaction requested=false 2023-03-29 03:55:49,692 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0337be0f939be2a2b593eefecee32758: 2023-03-29 03:55:49,692 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:49,692 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:49,692 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:49,692 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:49,692 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:49,692 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:49,694 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:49,694 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:49,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:49,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:49,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:49,694 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:49,694 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:49,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:49,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:49,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:49,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:49,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:49,696 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:49,696 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@41c0b117[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:49,696 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:49,696 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:49,698 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:49,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:49,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:49,698 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:49,698 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:49,698 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:49,698 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:49,698 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:49,699 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:49,699 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:49,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:49,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:49,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:49,699 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:49,700 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:49,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:49,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:49,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:49,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:49,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:49,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:49,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:49,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:49,702 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:49,702 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:49,702 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:49,702 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:49,702 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:49,702 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:49,704 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:49,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:49,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:49,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:49,704 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:49,704 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:49,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:49,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:49,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:49,706 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:49,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:49,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:49,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:49,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:49,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:49,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:49,707 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:49,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:49,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:49,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:49,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:49,712 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:49,712 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:49,713 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:49,712 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:49,713 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:49,713 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:49,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:49,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:49,713 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:49,713 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:49,713 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:55:49,713 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:49,713 DEBUG [Thread-3261] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:49,713 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:49,713 DEBUG [Thread-3261] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:49,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:49,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:49,713 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:49,713 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:49,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on 0337be0f939be2a2b593eefecee32758 2023-03-29 03:55:49,750 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 0337be0f939be2a2b593eefecee32758 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-03-29 03:55:49,750 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0337be0f939be2a2b593eefecee32758, store=A 2023-03-29 03:55:49,750 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:55:49,750 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0337be0f939be2a2b593eefecee32758, store=B 2023-03-29 03:55:49,750 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:55:49,750 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0337be0f939be2a2b593eefecee32758, store=C 2023-03-29 03:55:49,750 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:55:49,785 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:49,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42004 deadline: 1680062209780, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:49,787 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:49,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42010 deadline: 1680062209785, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:49,813 DEBUG [Thread-3261] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:49,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:49,815 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:49,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:49,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:49,816 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:49,816 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:49,817 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:49,817 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:49,817 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:49,817 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:49,818 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:49,818 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:49,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:49,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:49,818 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:49,818 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:49,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:49,819 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:49,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:49,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:49,819 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:49,819 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:49,819 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:49,819 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:49,819 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:49,820 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:49,820 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:49,820 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:49,820 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:49,820 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. started... 2023-03-29 03:55:49,820 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. as already flushing 2023-03-29 03:55:49,820 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:49,820 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:49,820 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:49,820 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:49,820 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:49,820 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:49,821 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:49,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:49,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:49,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:49,822 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:49,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:49,822 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:49,822 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:49,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:49,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:49,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:49,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:49,823 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:49,823 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@45cc1171[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:49,823 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:49,823 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:49,824 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:49,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:49,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:49,824 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:49,824 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:49,824 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:49,825 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:49,824 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:49,832 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:49,832 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:49,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:49,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:49,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:49,832 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:49,832 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:49,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:49,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:49,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:49,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:49,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:49,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:49,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:49,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:49,834 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:49,834 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:49,834 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:49,834 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:49,834 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:49,834 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:49,836 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:49,836 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:49,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:49,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:49,836 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:49,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:49,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:49,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:49,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:49,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:49,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:49,836 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:49,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:49,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:49,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:49,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:49,837 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:49,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:49,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:49,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:49,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:49,840 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:49,840 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:49,840 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:49,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:49,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:49,840 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:49,840 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:49,840 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:49,840 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:49,840 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:49,840 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:49,840 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:49,840 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:49,840 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:49,840 DEBUG [Thread-3261] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:49,840 DEBUG [Thread-3261] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:49,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:49,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:49,889 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:49,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42004 deadline: 1680062209886, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:49,891 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:49,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42010 deadline: 1680062209888, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:49,941 DEBUG [Thread-3261] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:49,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:49,943 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:49,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:49,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:49,944 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:49,944 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:49,944 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:49,944 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:49,944 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:49,944 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:49,945 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:49,945 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:49,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:49,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:49,946 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:49,946 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:49,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:49,946 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:49,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:49,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:49,946 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:49,946 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:49,947 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:49,947 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:49,947 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:49,947 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:49,947 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:49,947 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:49,947 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:49,947 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. started... 2023-03-29 03:55:49,947 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. as already flushing 2023-03-29 03:55:49,947 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:49,947 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:49,947 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:49,947 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:49,947 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:49,947 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:49,950 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:49,950 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:49,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:49,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:49,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:49,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:49,951 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:49,951 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:49,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:49,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:49,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:49,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:49,952 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:49,952 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@414ea4ed[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:49,952 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:49,952 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:49,953 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:49,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:49,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:49,953 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:49,953 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:49,953 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:49,953 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:49,953 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:49,955 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:49,955 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:49,955 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:49,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:49,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:49,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:49,955 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:49,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:49,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:49,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:49,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:49,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:49,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:49,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:49,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:49,957 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:49,957 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:49,957 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:49,957 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:49,957 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:49,957 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:49,958 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:49,958 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:49,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:49,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:49,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:49,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:49,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:49,958 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:49,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:49,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:49,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:49,959 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:49,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:49,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:49,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:49,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:49,960 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:49,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:49,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:49,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:49,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:49,963 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:49,963 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:49,963 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:49,963 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:49,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:49,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:49,963 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:49,963 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:49,963 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:49,963 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:49,963 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:49,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:49,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:49,964 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:49,964 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:49,964 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:49,965 DEBUG [Thread-3261] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:49,965 DEBUG [Thread-3261] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:50,065 DEBUG [Thread-3261] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:50,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:50,067 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:50,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:50,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:50,068 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:50,068 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:50,068 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:50,068 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:50,068 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:50,069 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:50,070 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:50,070 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:50,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:50,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:50,070 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:50,070 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:50,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:50,071 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:50,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:50,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:50,071 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:50,071 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:50,071 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:50,071 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:50,071 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:50,071 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:50,071 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:50,071 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:50,072 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:50,072 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. started... 2023-03-29 03:55:50,072 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. as already flushing 2023-03-29 03:55:50,072 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:50,072 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:50,072 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:50,072 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:50,072 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:50,072 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:50,073 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:50,073 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:50,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:50,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:50,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:50,074 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:50,074 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:50,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:50,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:50,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:50,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:50,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:50,075 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:50,075 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2bd2c2e6[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:50,075 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:50,075 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:50,076 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:50,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:50,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:50,077 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:50,077 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:50,077 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:50,077 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:50,077 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:50,080 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:50,080 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:50,080 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:50,080 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:50,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:50,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:50,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:50,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:50,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:50,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:50,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:50,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:50,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:50,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:50,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:50,082 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:50,082 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:50,082 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:50,082 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:50,082 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:50,082 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:50,084 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:50,084 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:50,084 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:50,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:50,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:50,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:50,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:50,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:50,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:50,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:50,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:50,084 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:50,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:50,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:50,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:50,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:50,086 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:50,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:50,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:50,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:50,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:50,088 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:50,088 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:50,088 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:50,088 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:50,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:50,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:50,088 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:50,088 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:50,088 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:50,088 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:50,088 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:50,088 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:55:50,088 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:50,088 DEBUG [Thread-3261] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:50,088 DEBUG [Thread-3261] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:50,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:50,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:50,088 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:50,088 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:50,095 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:50,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42004 deadline: 1680062210091, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:50,095 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:50,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42010 deadline: 1680062210092, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:50,157 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=119 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/A/4b7b48c3c92f4326a91c49a708364441 2023-03-29 03:55:50,188 DEBUG [Thread-3261] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:50,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:50,190 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:50,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:50,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:50,191 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:50,191 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:50,191 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:50,191 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:50,192 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:50,192 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:50,195 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:50,195 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:50,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:50,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:50,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:50,196 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:50,196 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:50,196 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:50,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:50,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:50,196 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:50,197 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:50,197 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:50,197 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:50,197 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:50,197 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:50,197 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:50,197 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:50,197 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:50,197 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. started... 2023-03-29 03:55:50,197 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. as already flushing 2023-03-29 03:55:50,197 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:50,197 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:50,197 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:50,197 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:50,197 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:50,197 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:50,199 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:50,199 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:50,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:50,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:50,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:50,199 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:50,199 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:50,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:50,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:50,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:50,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:50,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:50,200 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:50,200 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6fde9682[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:50,200 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:50,200 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:50,201 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:50,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:50,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:50,202 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:50,202 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:50,202 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:50,202 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:50,202 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:50,204 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:50,204 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:50,204 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:50,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:50,204 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:50,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:50,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:50,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:50,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:50,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:50,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:50,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:50,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:50,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:50,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:50,206 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:50,206 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:50,206 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:50,206 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:50,206 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:50,206 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:50,208 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:50,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:50,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:50,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:50,208 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:50,208 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:50,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:50,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:50,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:50,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:50,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:50,208 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:50,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:50,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:50,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:50,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:50,210 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:50,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:50,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:50,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:50,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:50,211 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:50,211 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:50,211 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:50,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:50,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:50,212 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:50,212 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:50,212 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:50,212 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:50,212 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:50,212 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:50,212 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:50,212 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:50,212 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:50,212 DEBUG [Thread-3261] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:50,212 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:55:50,212 DEBUG [Thread-3261] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:50,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:50,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:50,312 DEBUG [Thread-3261] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:50,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:50,314 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:50,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:50,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:50,315 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:50,315 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:50,315 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:50,315 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:50,316 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:50,316 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:50,317 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:50,317 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:50,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:50,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:50,317 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:50,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:50,317 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:50,318 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:50,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:50,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:50,318 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:50,318 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:50,318 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:50,318 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:50,318 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:50,318 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:50,319 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:50,319 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:50,319 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:50,319 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. started... 2023-03-29 03:55:50,319 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. as already flushing 2023-03-29 03:55:50,319 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:50,319 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:50,319 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:50,319 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:50,319 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:50,319 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:50,322 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:50,322 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:50,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:50,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:50,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:50,322 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:50,322 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:50,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:50,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:50,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:50,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:50,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:50,324 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:50,324 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a3c5591[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:50,324 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:50,324 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:50,325 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:50,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:50,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:50,325 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:50,325 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:50,325 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:50,325 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:50,325 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:50,327 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:50,327 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:50,327 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:50,327 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:50,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:50,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:50,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:50,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:50,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:50,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:50,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:50,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:50,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:50,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:50,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:50,330 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:50,330 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:50,330 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:50,330 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:50,330 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:50,330 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:50,332 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:50,332 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:50,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:50,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:50,332 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:50,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:50,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:50,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:50,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:50,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:50,332 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:50,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:50,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:50,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:50,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:50,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:50,335 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:50,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:50,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:50,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:50,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:50,339 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:50,339 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:50,339 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:50,339 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:50,339 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:50,339 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:50,339 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:50,339 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:50,339 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:50,339 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:50,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:50,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:50,339 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:50,339 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:50,339 DEBUG [Thread-3261] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:50,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:50,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:50,339 DEBUG [Thread-3261] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:50,397 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:50,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42004 deadline: 1680062210396, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:50,397 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:50,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42010 deadline: 1680062210397, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:50,440 DEBUG [Thread-3261] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:50,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:50,443 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:50,446 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:50,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42024 deadline: 1680062210441, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:50,449 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:50,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41942 deadline: 1680062210443, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:50,454 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:50,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41960 deadline: 1680062210448, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:50,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:50,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:50,456 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:50,456 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:50,456 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:50,456 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:50,458 DEBUG [Thread-3253] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4149 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758., hostname=jenkins-hbase4.apache.org,35599,1680062018534, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-29 03:55:50,458 DEBUG [Thread-3257] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4154 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758., hostname=jenkins-hbase4.apache.org,35599,1680062018534, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-29 03:55:50,458 DEBUG [Thread-3251] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4156 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758., hostname=jenkins-hbase4.apache.org,35599,1680062018534, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-29 03:55:50,458 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:50,458 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:50,460 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:50,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:50,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:50,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:50,460 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:50,461 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:50,461 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:50,461 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:50,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:50,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:50,461 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:50,461 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:50,461 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:50,461 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:50,462 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:50,462 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:50,468 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:50,468 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:50,468 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:50,468 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. started... 2023-03-29 03:55:50,468 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. as already flushing 2023-03-29 03:55:50,468 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:50,468 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:50,468 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:50,468 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:50,468 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:50,468 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:50,470 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:50,470 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:50,470 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:50,471 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:50,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:50,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:50,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:50,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:50,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:50,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:50,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:50,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:50,473 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:50,473 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7e61f9f7[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:50,473 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:50,473 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:50,474 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:50,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:50,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:50,475 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:50,475 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:50,475 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:50,477 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:50,477 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:50,477 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:50,477 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:50,478 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:50,478 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:50,478 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:50,478 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:50,478 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:50,478 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:50,478 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:50,480 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:50,480 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:50,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:50,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:50,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:50,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:50,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:50,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:50,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:50,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:50,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:50,480 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:50,481 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:50,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:50,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:50,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:50,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:50,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:50,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:50,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:50,484 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:50,503 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:50,503 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:50,503 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:50,503 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:50,503 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:50,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:50,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:50,503 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:50,503 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:50,503 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:50,503 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:50,503 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:50,503 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:50,503 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:50,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:50,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:50,504 DEBUG [Thread-3261] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:50,504 DEBUG [Thread-3261] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:50,569 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=119 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/B/29ee69f7d34248348ca914a13a0b6928 2023-03-29 03:55:50,587 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=119 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/C/82d6e07a2f4b4661a7dfd3be6f7bc65b 2023-03-29 03:55:50,590 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/A/4b7b48c3c92f4326a91c49a708364441 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/4b7b48c3c92f4326a91c49a708364441 2023-03-29 03:55:50,594 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/4b7b48c3c92f4326a91c49a708364441, entries=200, sequenceid=119, filesize=13.9 K 2023-03-29 03:55:50,595 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/B/29ee69f7d34248348ca914a13a0b6928 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/29ee69f7d34248348ca914a13a0b6928 2023-03-29 03:55:50,602 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/29ee69f7d34248348ca914a13a0b6928, entries=150, sequenceid=119, filesize=11.6 K 2023-03-29 03:55:50,603 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/C/82d6e07a2f4b4661a7dfd3be6f7bc65b as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/82d6e07a2f4b4661a7dfd3be6f7bc65b 2023-03-29 03:55:50,604 DEBUG [Thread-3261] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:50,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:50,607 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:50,610 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/82d6e07a2f4b4661a7dfd3be6f7bc65b, entries=150, sequenceid=119, filesize=11.6 K 2023-03-29 03:55:50,611 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 0337be0f939be2a2b593eefecee32758 in 861ms, sequenceid=119, compaction requested=true 2023-03-29 03:55:50,611 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 0337be0f939be2a2b593eefecee32758: 2023-03-29 03:55:50,611 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:55:50,611 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-29 03:55:50,611 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-29 03:55:50,611 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:55:50,612 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:55:50,612 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 38204 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-29 03:55:50,612 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1912): 0337be0f939be2a2b593eefecee32758/A is initiating minor compaction (all files) 2023-03-29 03:55:50,613 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 0337be0f939be2a2b593eefecee32758/A in TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:50,613 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/1416602e3e27463ab58f12fe33aeed04, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/400588da025049dbaaeb6f7e5d267f09, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/4b7b48c3c92f4326a91c49a708364441] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp, totalSize=37.3 K 2023-03-29 03:55:50,613 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35864 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-29 03:55:50,613 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1912): 0337be0f939be2a2b593eefecee32758/B is initiating minor compaction (all files) 2023-03-29 03:55:50,613 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0337be0f939be2a2b593eefecee32758/B in TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:50,613 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/ff38438e894640a9a40125a321c52993, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/f77132a4406e418bb8ea6dc33ec073d1, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/29ee69f7d34248348ca914a13a0b6928] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp, totalSize=35.0 K 2023-03-29 03:55:50,613 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 1416602e3e27463ab58f12fe33aeed04, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1680062148123 2023-03-29 03:55:50,613 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting ff38438e894640a9a40125a321c52993, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1680062148123 2023-03-29 03:55:50,613 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 400588da025049dbaaeb6f7e5d267f09, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=102, earliestPutTs=1680062148783 2023-03-29 03:55:50,614 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting f77132a4406e418bb8ea6dc33ec073d1, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=102, earliestPutTs=1680062148783 2023-03-29 03:55:50,614 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 4b7b48c3c92f4326a91c49a708364441, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=119, earliestPutTs=1680062149432 2023-03-29 03:55:50,614 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 29ee69f7d34248348ca914a13a0b6928, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=119, earliestPutTs=1680062149437 2023-03-29 03:55:50,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:50,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:50,616 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:50,616 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:50,616 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:50,616 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:50,616 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:50,616 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:50,618 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:50,618 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:50,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:50,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:50,618 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:50,618 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:50,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:50,619 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:50,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:50,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:50,619 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:50,628 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:50,628 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:50,628 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:50,628 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:50,645 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] throttle.PressureAwareThroughputController(145): 0337be0f939be2a2b593eefecee32758#B#compaction#454 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:55:50,653 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0337be0f939be2a2b593eefecee32758#A#compaction#455 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:55:50,668 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:50,668 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:50,668 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:50,668 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. started... 2023-03-29 03:55:50,668 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0337be0f939be2a2b593eefecee32758 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-03-29 03:55:50,669 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0337be0f939be2a2b593eefecee32758, store=A 2023-03-29 03:55:50,669 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:55:50,669 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0337be0f939be2a2b593eefecee32758, store=B 2023-03-29 03:55:50,669 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:55:50,669 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0337be0f939be2a2b593eefecee32758, store=C 2023-03-29 03:55:50,669 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:55:50,723 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/A/557ebd52497442e89b55a5ef6402788e as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/557ebd52497442e89b55a5ef6402788e 2023-03-29 03:55:50,729 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0337be0f939be2a2b593eefecee32758/A of 0337be0f939be2a2b593eefecee32758 into 557ebd52497442e89b55a5ef6402788e(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:55:50,729 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0337be0f939be2a2b593eefecee32758: 2023-03-29 03:55:50,729 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758., storeName=0337be0f939be2a2b593eefecee32758/A, priority=13, startTime=1680062150611; duration=0sec 2023-03-29 03:55:50,729 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:55:50,729 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-29 03:55:50,730 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35864 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-29 03:55:50,730 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1912): 0337be0f939be2a2b593eefecee32758/C is initiating minor compaction (all files) 2023-03-29 03:55:50,730 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 0337be0f939be2a2b593eefecee32758/C in TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:50,731 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/9a96ecf7a8134fce99f2d5c33f57647f, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/c29e41d57b164baa98de1a169f6a6597, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/82d6e07a2f4b4661a7dfd3be6f7bc65b] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp, totalSize=35.0 K 2023-03-29 03:55:50,731 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 9a96ecf7a8134fce99f2d5c33f57647f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1680062148123 2023-03-29 03:55:50,731 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting c29e41d57b164baa98de1a169f6a6597, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=102, earliestPutTs=1680062148783 2023-03-29 03:55:50,732 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 82d6e07a2f4b4661a7dfd3be6f7bc65b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=119, earliestPutTs=1680062149437 2023-03-29 03:55:50,736 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=135 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/A/6208b63e83a2467fab77a17ffd027841 2023-03-29 03:55:50,749 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0337be0f939be2a2b593eefecee32758#C#compaction#457 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:55:50,776 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/C/0fad591a5f4c4b3092427eb05ba2a9df as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/0fad591a5f4c4b3092427eb05ba2a9df 2023-03-29 03:55:50,782 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0337be0f939be2a2b593eefecee32758/C of 0337be0f939be2a2b593eefecee32758 into 0fad591a5f4c4b3092427eb05ba2a9df(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:55:50,782 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0337be0f939be2a2b593eefecee32758: 2023-03-29 03:55:50,782 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758., storeName=0337be0f939be2a2b593eefecee32758/C, priority=13, startTime=1680062150612; duration=0sec 2023-03-29 03:55:50,782 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:55:50,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on 0337be0f939be2a2b593eefecee32758 2023-03-29 03:55:50,907 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. as already flushing 2023-03-29 03:55:50,966 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:50,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42004 deadline: 1680062210961, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:50,971 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:50,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42010 deadline: 1680062210962, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:51,073 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:51,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42004 deadline: 1680062211067, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:51,076 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:51,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42010 deadline: 1680062211073, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:51,102 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/B/0233060e9cbe4c188069a55788552a3e as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/0233060e9cbe4c188069a55788552a3e 2023-03-29 03:55:51,107 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0337be0f939be2a2b593eefecee32758/B of 0337be0f939be2a2b593eefecee32758 into 0233060e9cbe4c188069a55788552a3e(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:55:51,107 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0337be0f939be2a2b593eefecee32758: 2023-03-29 03:55:51,107 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758., storeName=0337be0f939be2a2b593eefecee32758/B, priority=13, startTime=1680062150611; duration=0sec 2023-03-29 03:55:51,107 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:55:51,171 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=135 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/B/9d4f50edeef7455cb4751971c6177661 2023-03-29 03:55:51,203 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=135 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/C/a515905a06bd485e9634b6af7713c07b 2023-03-29 03:55:51,211 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/A/6208b63e83a2467fab77a17ffd027841 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/6208b63e83a2467fab77a17ffd027841 2023-03-29 03:55:51,216 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/6208b63e83a2467fab77a17ffd027841, entries=150, sequenceid=135, filesize=11.8 K 2023-03-29 03:55:51,217 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/B/9d4f50edeef7455cb4751971c6177661 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/9d4f50edeef7455cb4751971c6177661 2023-03-29 03:55:51,221 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/9d4f50edeef7455cb4751971c6177661, entries=150, sequenceid=135, filesize=11.8 K 2023-03-29 03:55:51,223 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/C/a515905a06bd485e9634b6af7713c07b as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/a515905a06bd485e9634b6af7713c07b 2023-03-29 03:55:51,228 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/a515905a06bd485e9634b6af7713c07b, entries=150, sequenceid=135, filesize=11.8 K 2023-03-29 03:55:51,229 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 0337be0f939be2a2b593eefecee32758 in 561ms, sequenceid=135, compaction requested=false 2023-03-29 03:55:51,230 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0337be0f939be2a2b593eefecee32758: 2023-03-29 03:55:51,230 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:51,230 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:51,230 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:51,230 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:51,232 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:51,232 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:51,236 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:51,237 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:51,237 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:51,238 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:51,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:51,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:51,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:51,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:51,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:51,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:51,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:51,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:51,240 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:51,240 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4399c12f[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:51,240 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:51,240 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:51,243 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:51,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:51,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:51,243 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:51,243 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:51,243 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:51,243 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:51,243 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:51,244 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:51,244 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:51,244 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:51,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:51,244 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:51,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:51,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:51,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:51,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:51,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:51,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:51,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:51,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:51,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:51,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:51,249 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:51,249 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:51,249 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:51,249 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:51,249 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:51,249 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:51,252 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:51,252 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:51,252 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:51,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:51,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:51,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:51,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:51,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:51,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:51,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:51,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:51,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:51,253 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:51,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:51,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:51,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:51,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:51,255 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:51,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:51,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:51,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:51,257 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:51,258 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:51,258 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:51,258 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:51,258 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:51,258 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:51,258 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:51,258 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:55:51,258 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:51,258 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:51,258 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:51,258 DEBUG [Thread-3261] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:51,258 DEBUG [Thread-3261] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:51,258 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:51,258 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:51,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:51,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:51,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:51,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:51,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on 0337be0f939be2a2b593eefecee32758 2023-03-29 03:55:51,278 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 0337be0f939be2a2b593eefecee32758 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-03-29 03:55:51,279 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0337be0f939be2a2b593eefecee32758, store=A 2023-03-29 03:55:51,279 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:55:51,279 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0337be0f939be2a2b593eefecee32758, store=B 2023-03-29 03:55:51,279 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:55:51,279 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0337be0f939be2a2b593eefecee32758, store=C 2023-03-29 03:55:51,279 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:55:51,329 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:51,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42010 deadline: 1680062211320, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:51,329 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:51,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42004 deadline: 1680062211324, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:51,358 DEBUG [Thread-3261] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:51,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:51,361 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:51,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:51,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:51,362 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:51,362 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:51,362 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:51,362 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:51,362 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:51,362 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:51,364 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:51,364 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:51,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:51,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:51,364 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:51,364 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:51,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:51,365 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:51,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:51,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:51,365 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:51,366 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:51,366 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:51,366 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:51,366 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:51,366 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:51,374 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:51,374 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:51,374 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:51,375 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. started... 2023-03-29 03:55:51,375 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. as already flushing 2023-03-29 03:55:51,375 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:51,375 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:51,375 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:51,375 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:51,375 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:51,375 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:51,378 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:51,378 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:51,378 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:51,384 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:51,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:51,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:51,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:51,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:51,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:51,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:51,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:51,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:51,387 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:51,387 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@58e82558[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:51,387 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:51,387 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:51,389 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:51,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:51,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:51,389 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:51,389 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:51,389 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:51,390 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:51,390 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:51,392 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:51,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:51,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:51,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:51,392 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:51,392 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:51,392 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:51,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:51,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:51,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:51,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:51,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:51,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:51,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:51,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:51,395 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:51,396 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:51,396 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:51,396 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:51,396 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:51,396 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:51,398 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:51,398 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:51,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:51,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:51,398 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:51,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:51,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:51,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:51,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:51,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:51,399 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:51,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:51,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:51,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:51,403 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:51,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:51,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:51,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:51,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:51,406 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:51,406 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:51,406 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:51,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:51,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:51,407 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:51,407 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:51,407 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:51,407 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:51,407 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:51,407 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:51,407 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:51,407 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:51,407 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:51,407 DEBUG [Thread-3261] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:51,407 DEBUG [Thread-3261] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:51,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:51,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:51,408 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:55:51,437 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:51,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42010 deadline: 1680062211430, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:51,438 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:51,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42004 deadline: 1680062211430, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:51,507 DEBUG [Thread-3261] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:51,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:51,509 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:51,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:51,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:51,510 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:51,510 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:51,510 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:51,510 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:51,511 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:51,511 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:51,512 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:51,512 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:51,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:51,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:51,513 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:51,513 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:51,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:51,513 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:51,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:51,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:51,513 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:51,514 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:51,514 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:51,514 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:51,514 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:51,514 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:51,514 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:51,514 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:51,514 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:51,514 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. started... 2023-03-29 03:55:51,514 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. as already flushing 2023-03-29 03:55:51,514 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:51,514 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:51,514 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:51,514 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:51,514 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:51,514 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:51,516 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:51,516 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:51,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:51,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:51,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:51,516 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:51,516 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:51,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:51,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:51,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:51,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:51,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:51,519 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:51,519 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70d06309[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:51,519 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:51,519 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:51,520 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:51,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:51,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:51,520 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:51,520 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:51,520 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:51,520 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:51,520 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:51,523 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:51,523 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:51,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:51,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:51,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:51,523 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:51,523 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:51,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:51,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:51,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:51,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:51,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:51,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:51,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:51,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:51,526 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:51,526 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:51,526 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:51,526 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:51,526 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:51,526 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:51,529 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:51,529 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:51,529 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:51,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:51,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:51,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:51,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:51,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:51,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:51,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:51,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:51,529 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:51,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:51,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:51,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:51,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:51,532 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:51,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:51,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:51,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:51,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:51,534 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:51,534 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:51,534 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:51,534 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:51,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:51,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:51,534 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:51,534 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:51,534 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:51,534 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:51,534 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:51,535 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:51,535 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:51,535 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:51,535 DEBUG [Thread-3261] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:51,535 DEBUG [Thread-3261] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:51,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:51,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:51,635 DEBUG [Thread-3261] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:51,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:51,638 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:51,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:51,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:51,639 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:51,639 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:51,639 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:51,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42004 deadline: 1680062211638, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:51,639 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:51,639 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:51,640 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:51,640 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:51,641 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:51,641 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:51,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:51,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:51,641 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:51,641 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:51,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:51,642 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:51,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:51,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:51,642 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:51,642 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:51,642 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:51,642 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:51,642 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:51,642 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:51,643 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:51,643 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:51,643 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:51,643 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. started... 2023-03-29 03:55:51,643 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. as already flushing 2023-03-29 03:55:51,643 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:51,643 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:51,643 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:51,643 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:51,643 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:51,643 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:51,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42010 deadline: 1680062211641, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:51,643 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:51,646 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:51,646 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:51,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:51,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:51,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:51,646 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:51,646 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:51,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:51,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:51,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:51,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:51,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:51,648 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:51,648 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@10f92a5[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:51,648 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:51,648 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:51,649 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:51,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:51,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:51,649 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:51,649 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:51,649 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:51,649 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:51,649 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:51,653 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:51,653 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:51,654 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:51,654 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:51,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:51,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:51,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:51,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:51,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:51,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:51,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:51,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:51,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:51,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:51,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:51,656 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:51,656 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:51,656 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:51,656 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:51,656 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:51,656 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:51,657 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:51,657 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:51,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:51,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:51,657 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:51,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:51,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:51,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:51,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:51,658 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:51,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:51,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:51,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:51,659 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:51,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:51,661 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:51,661 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:51,661 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:51,661 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:51,661 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:51,661 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:51,661 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:51,661 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:51,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:51,661 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:51,661 DEBUG [Thread-3261] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:51,661 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:55:51,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:51,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:51,661 DEBUG [Thread-3261] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:51,661 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:51,661 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:51,662 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:51,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:51,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:51,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:51,663 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:55:51,710 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=155 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/A/d7a37ee0bd1d46ebbeb7d45610d8bf92 2023-03-29 03:55:51,762 DEBUG [Thread-3261] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:51,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:51,764 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:51,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:51,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:51,765 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:51,765 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:51,765 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:51,765 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:51,765 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:51,765 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:51,767 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:51,767 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:51,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:51,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:51,767 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:51,767 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:51,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:51,768 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:51,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:51,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:51,768 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:51,768 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:51,768 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:51,769 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:51,769 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:51,769 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:51,769 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:51,769 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:51,769 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:51,769 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. started... 2023-03-29 03:55:51,769 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. as already flushing 2023-03-29 03:55:51,769 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:51,769 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:51,769 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:51,769 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:51,769 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:51,769 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:51,771 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:51,771 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:51,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:51,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:51,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:51,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:51,771 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:51,771 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:51,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:51,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:51,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:51,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:51,773 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:51,773 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@c12ab12[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:51,773 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:51,773 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:51,774 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:51,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:51,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:51,774 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:51,774 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:51,774 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:51,774 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:51,774 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:51,775 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:51,775 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:51,775 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:51,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:51,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:51,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:51,775 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:51,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:51,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:51,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:51,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:51,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:51,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:51,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:51,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:51,778 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:51,778 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:51,778 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:51,778 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:51,778 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:51,778 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:51,781 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:51,781 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:51,781 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:51,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:51,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:51,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:51,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:51,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:51,781 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:51,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:51,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:51,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:51,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:51,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:51,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:51,783 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:51,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:51,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:51,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:51,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:51,785 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:51,785 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:51,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:51,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:51,786 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:51,786 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:51,786 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:51,786 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:51,786 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:51,786 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:51,786 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:51,786 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:51,786 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:51,786 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:51,786 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:55:51,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:51,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:51,786 DEBUG [Thread-3261] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:51,786 DEBUG [Thread-3261] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:51,886 DEBUG [Thread-3261] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:51,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:51,888 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:51,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:51,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:51,889 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:51,889 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:51,889 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:51,889 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:51,890 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:51,890 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:51,891 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:51,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:51,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:51,891 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:51,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:51,892 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:51,892 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:51,892 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:51,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:51,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:51,893 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:51,893 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:51,893 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:51,893 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:51,893 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:51,893 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:51,893 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:51,894 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:51,894 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:51,894 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. started... 2023-03-29 03:55:51,894 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. as already flushing 2023-03-29 03:55:51,894 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:51,894 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:51,894 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:51,894 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:51,894 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:51,894 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:51,896 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:51,896 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:51,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:51,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:51,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:51,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:51,896 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:51,896 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:51,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:51,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:51,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:51,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:51,899 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:51,899 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@15c98d2d[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:51,899 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:51,899 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:51,901 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:51,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:51,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:51,901 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:51,901 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:51,901 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:51,901 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:51,901 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:51,902 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:51,903 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:51,903 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:51,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:51,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:51,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:51,903 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:51,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:51,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:51,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:51,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:51,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:51,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:51,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:51,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:51,905 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:51,905 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:51,905 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:51,905 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:51,905 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:51,905 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:51,907 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:51,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:51,907 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:51,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:51,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:51,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:51,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:51,907 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:51,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:51,907 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:51,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:51,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:51,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:51,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:51,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:51,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:51,909 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:51,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:51,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:51,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:51,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:51,911 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:51,911 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:51,911 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:51,911 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:51,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:51,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:51,911 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:51,911 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:51,911 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:51,911 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:51,911 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:51,911 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:51,911 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:51,911 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:51,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:51,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:51,911 DEBUG [Thread-3261] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:51,911 DEBUG [Thread-3261] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:51,945 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:51,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42004 deadline: 1680062211940, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:51,948 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:51,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42010 deadline: 1680062211944, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:52,011 DEBUG [Thread-3261] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:52,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:52,016 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:52,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:52,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:52,020 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:52,020 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:52,020 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:52,020 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:52,021 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:52,021 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:52,023 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:52,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:52,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:52,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:52,023 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:52,024 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:52,024 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:52,024 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:52,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:52,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:52,024 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:52,024 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:52,024 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:52,024 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:52,024 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:52,024 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:52,028 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:52,028 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:52,028 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:52,028 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. started... 2023-03-29 03:55:52,028 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. as already flushing 2023-03-29 03:55:52,028 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:52,028 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:52,028 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:52,028 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:52,028 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:52,028 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:52,030 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:52,030 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:52,030 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:52,031 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:52,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:52,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:52,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:52,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:52,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:52,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:52,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:52,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:52,032 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:52,033 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:52,033 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@66726cb6[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:52,033 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:52,034 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:52,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:52,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:52,034 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:52,034 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:52,034 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:52,035 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:52,035 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:52,035 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:52,035 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:52,036 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:52,036 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:52,036 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:52,036 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:52,036 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:52,036 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:52,036 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:52,043 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:52,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:52,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:52,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:52,043 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:52,043 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:52,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:52,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:52,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:52,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:52,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:52,044 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:52,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:52,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:52,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:52,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:52,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:52,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:52,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:52,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:52,049 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:52,055 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:52,055 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:52,055 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:52,055 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:52,055 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:52,055 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:52,055 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:52,055 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:52,056 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:52,056 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:52,056 DEBUG [Thread-3261] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:52,056 DEBUG [Thread-3261] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:52,063 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:52,063 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:52,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:52,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:52,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:52,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:52,120 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=155 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/B/19a197899dd3467b9f5599e1c6d707bc 2023-03-29 03:55:52,155 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=155 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/C/23bbbea184ec4585943bac033c88f5ac 2023-03-29 03:55:52,156 DEBUG [Thread-3261] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:52,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:52,158 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:52,159 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/A/d7a37ee0bd1d46ebbeb7d45610d8bf92 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/d7a37ee0bd1d46ebbeb7d45610d8bf92 2023-03-29 03:55:52,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:52,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:52,160 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:52,160 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:52,160 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:52,160 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:52,160 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:52,160 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:52,161 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:52,161 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:52,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:52,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:52,162 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:52,162 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:52,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:52,162 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:52,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:52,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:52,163 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:52,163 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:52,163 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:52,163 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:52,163 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:52,163 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:52,165 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/d7a37ee0bd1d46ebbeb7d45610d8bf92, entries=200, sequenceid=155, filesize=14.1 K 2023-03-29 03:55:52,166 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:52,166 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:52,166 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:52,166 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. started... 2023-03-29 03:55:52,166 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. as already flushing 2023-03-29 03:55:52,166 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:52,166 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:52,167 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:52,167 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:52,167 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:52,167 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/B/19a197899dd3467b9f5599e1c6d707bc as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/19a197899dd3467b9f5599e1c6d707bc 2023-03-29 03:55:52,167 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:52,169 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:52,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:52,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:52,169 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:52,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:52,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:52,170 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:52,170 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:52,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:52,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:52,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:52,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:52,171 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/19a197899dd3467b9f5599e1c6d707bc, entries=150, sequenceid=155, filesize=11.8 K 2023-03-29 03:55:52,171 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:52,171 DEBUG [MemStoreFlusher.0] regionserver.StoreScanner(953): StoreScanner already has the close lock. There is no need to updateReaders 2023-03-29 03:55:52,171 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:52,171 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:52,171 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b154067[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:52,172 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/C/23bbbea184ec4585943bac033c88f5ac as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/23bbbea184ec4585943bac033c88f5ac 2023-03-29 03:55:52,174 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:52,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:52,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:52,174 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:52,174 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:52,174 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:52,174 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:52,174 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:52,176 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:52,176 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:52,176 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:52,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:52,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:52,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:52,176 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:52,176 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/23bbbea184ec4585943bac033c88f5ac, entries=150, sequenceid=155, filesize=11.8 K 2023-03-29 03:55:52,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:52,177 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 0337be0f939be2a2b593eefecee32758 in 899ms, sequenceid=155, compaction requested=true 2023-03-29 03:55:52,177 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 0337be0f939be2a2b593eefecee32758: 2023-03-29 03:55:52,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:52,177 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-29 03:55:52,177 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:55:52,178 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:55:52,178 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-29 03:55:52,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:52,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:52,178 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-29 03:55:52,178 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 38656 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-29 03:55:52,179 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1912): 0337be0f939be2a2b593eefecee32758/A is initiating minor compaction (all files) 2023-03-29 03:55:52,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:52,179 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 0337be0f939be2a2b593eefecee32758/A in TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:52,179 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/557ebd52497442e89b55a5ef6402788e, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/6208b63e83a2467fab77a17ffd027841, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/d7a37ee0bd1d46ebbeb7d45610d8bf92] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp, totalSize=37.8 K 2023-03-29 03:55:52,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:52,179 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 557ebd52497442e89b55a5ef6402788e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=119, earliestPutTs=1680062149437 2023-03-29 03:55:52,180 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 6208b63e83a2467fab77a17ffd027841, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=135, earliestPutTs=1680062149774 2023-03-29 03:55:52,180 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting d7a37ee0bd1d46ebbeb7d45610d8bf92, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=155, earliestPutTs=1680062150939 2023-03-29 03:55:52,180 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36266 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-29 03:55:52,181 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1912): 0337be0f939be2a2b593eefecee32758/B is initiating minor compaction (all files) 2023-03-29 03:55:52,181 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0337be0f939be2a2b593eefecee32758/B in TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:52,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:52,181 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/0233060e9cbe4c188069a55788552a3e, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/9d4f50edeef7455cb4751971c6177661, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/19a197899dd3467b9f5599e1c6d707bc] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp, totalSize=35.4 K 2023-03-29 03:55:52,181 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 0233060e9cbe4c188069a55788552a3e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=119, earliestPutTs=1680062149437 2023-03-29 03:55:52,181 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 9d4f50edeef7455cb4751971c6177661, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=135, earliestPutTs=1680062149774 2023-03-29 03:55:52,182 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 19a197899dd3467b9f5599e1c6d707bc, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=155, earliestPutTs=1680062150959 2023-03-29 03:55:52,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:52,183 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:52,183 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:52,183 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:52,183 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:52,183 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:52,183 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:52,185 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:52,185 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:52,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:52,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:52,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:52,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:52,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:52,185 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:52,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:52,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:52,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:52,186 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:52,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:52,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:52,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:52,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:52,188 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0337be0f939be2a2b593eefecee32758#A#compaction#463 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:55:52,189 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:52,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:52,189 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] throttle.PressureAwareThroughputController(145): 0337be0f939be2a2b593eefecee32758#B#compaction#464 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:55:52,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:52,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:52,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:52,198 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:52,198 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:52,198 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:52,198 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:52,198 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:52,198 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:52,198 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:52,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:52,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:52,198 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:52,198 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:52,198 DEBUG [Thread-3261] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:52,198 DEBUG [Thread-3261] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:52,198 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:52,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:52,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:52,199 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:52,199 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:52,208 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/B/29eb1917b4d841779e0f57359d1101f0 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/29eb1917b4d841779e0f57359d1101f0 2023-03-29 03:55:52,212 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0337be0f939be2a2b593eefecee32758/B of 0337be0f939be2a2b593eefecee32758 into 29eb1917b4d841779e0f57359d1101f0(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:55:52,212 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0337be0f939be2a2b593eefecee32758: 2023-03-29 03:55:52,212 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758., storeName=0337be0f939be2a2b593eefecee32758/B, priority=13, startTime=1680062152178; duration=0sec 2023-03-29 03:55:52,212 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:55:52,212 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-29 03:55:52,213 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/A/462210eb463f48a793a5da760b1f06c4 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/462210eb463f48a793a5da760b1f06c4 2023-03-29 03:55:52,213 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36266 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-29 03:55:52,213 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1912): 0337be0f939be2a2b593eefecee32758/C is initiating minor compaction (all files) 2023-03-29 03:55:52,213 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0337be0f939be2a2b593eefecee32758/C in TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:52,214 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/0fad591a5f4c4b3092427eb05ba2a9df, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/a515905a06bd485e9634b6af7713c07b, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/23bbbea184ec4585943bac033c88f5ac] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp, totalSize=35.4 K 2023-03-29 03:55:52,214 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 0fad591a5f4c4b3092427eb05ba2a9df, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=119, earliestPutTs=1680062149437 2023-03-29 03:55:52,214 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting a515905a06bd485e9634b6af7713c07b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=135, earliestPutTs=1680062149774 2023-03-29 03:55:52,215 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 23bbbea184ec4585943bac033c88f5ac, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=155, earliestPutTs=1680062150959 2023-03-29 03:55:52,218 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0337be0f939be2a2b593eefecee32758/A of 0337be0f939be2a2b593eefecee32758 into 462210eb463f48a793a5da760b1f06c4(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:55:52,218 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0337be0f939be2a2b593eefecee32758: 2023-03-29 03:55:52,218 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758., storeName=0337be0f939be2a2b593eefecee32758/A, priority=13, startTime=1680062152177; duration=0sec 2023-03-29 03:55:52,218 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:55:52,221 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] throttle.PressureAwareThroughputController(145): 0337be0f939be2a2b593eefecee32758#C#compaction#465 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:55:52,241 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/C/bf79062a472342099e8940b53d90c888 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/bf79062a472342099e8940b53d90c888 2023-03-29 03:55:52,247 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0337be0f939be2a2b593eefecee32758/C of 0337be0f939be2a2b593eefecee32758 into bf79062a472342099e8940b53d90c888(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:55:52,247 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0337be0f939be2a2b593eefecee32758: 2023-03-29 03:55:52,247 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758., storeName=0337be0f939be2a2b593eefecee32758/C, priority=13, startTime=1680062152178; duration=0sec 2023-03-29 03:55:52,247 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:55:52,299 DEBUG [Thread-3261] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:52,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:52,300 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:52,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:52,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:52,302 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:52,302 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:52,302 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:52,302 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:52,303 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:52,303 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:52,304 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:52,304 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:52,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:52,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:52,304 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:52,304 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:52,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:52,305 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:52,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:52,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:52,305 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:52,305 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:52,305 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:52,305 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:52,305 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:52,305 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:52,305 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:52,305 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:52,305 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:52,306 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. started... 2023-03-29 03:55:52,306 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0337be0f939be2a2b593eefecee32758 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-03-29 03:55:52,306 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0337be0f939be2a2b593eefecee32758, store=A 2023-03-29 03:55:52,306 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:55:52,306 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0337be0f939be2a2b593eefecee32758, store=B 2023-03-29 03:55:52,306 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:55:52,306 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0337be0f939be2a2b593eefecee32758, store=C 2023-03-29 03:55:52,306 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:55:52,315 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=173 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/A/7b242d42d4e6411b9bb939897c7fb883 2023-03-29 03:55:52,360 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=173 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/B/5b0d803ed6f54678bb227f66581a9faf 2023-03-29 03:55:52,372 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=173 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/C/fb54b30d187644b9abcd14be0da33059 2023-03-29 03:55:52,375 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/A/7b242d42d4e6411b9bb939897c7fb883 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/7b242d42d4e6411b9bb939897c7fb883 2023-03-29 03:55:52,378 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/7b242d42d4e6411b9bb939897c7fb883, entries=150, sequenceid=173, filesize=11.8 K 2023-03-29 03:55:52,379 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/B/5b0d803ed6f54678bb227f66581a9faf as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/5b0d803ed6f54678bb227f66581a9faf 2023-03-29 03:55:52,382 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/5b0d803ed6f54678bb227f66581a9faf, entries=150, sequenceid=173, filesize=11.8 K 2023-03-29 03:55:52,382 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/C/fb54b30d187644b9abcd14be0da33059 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/fb54b30d187644b9abcd14be0da33059 2023-03-29 03:55:52,386 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/fb54b30d187644b9abcd14be0da33059, entries=150, sequenceid=173, filesize=11.8 K 2023-03-29 03:55:52,387 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=0 B/0 for 0337be0f939be2a2b593eefecee32758 in 81ms, sequenceid=173, compaction requested=false 2023-03-29 03:55:52,387 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0337be0f939be2a2b593eefecee32758: 2023-03-29 03:55:52,387 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:52,387 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:52,387 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:52,387 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:52,387 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:52,387 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:52,391 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:52,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:52,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:52,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:52,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:52,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:52,392 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:52,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:52,393 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:52,393 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:52,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:52,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:52,394 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:52,394 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c6222c[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:52,394 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:52,394 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:52,395 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:52,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:52,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:52,396 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:52,396 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:52,396 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:52,396 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:52,396 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:52,398 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:52,398 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:52,398 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:52,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:52,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:52,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:52,398 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:52,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:52,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:52,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:52,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:52,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:52,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:52,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:52,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:52,401 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:52,401 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:52,401 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:52,401 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:52,401 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:52,401 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:52,402 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:52,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:52,403 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:52,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:52,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:52,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:52,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:52,403 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:52,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:52,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:52,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:52,403 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:52,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:52,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:52,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:52,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:52,404 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:52,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:52,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:52,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:52,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:52,407 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:52,407 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:52,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:52,407 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:52,407 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:52,407 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:52,407 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:52,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:52,407 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:52,408 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:52,408 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:55:52,408 DEBUG [Thread-3261] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:52,408 DEBUG [Thread-3261] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:52,408 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:52,408 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:52,408 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:52,408 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:52,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:52,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:52,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on 0337be0f939be2a2b593eefecee32758 2023-03-29 03:55:52,460 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 0337be0f939be2a2b593eefecee32758 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-29 03:55:52,461 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0337be0f939be2a2b593eefecee32758, store=A 2023-03-29 03:55:52,461 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:55:52,461 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0337be0f939be2a2b593eefecee32758, store=B 2023-03-29 03:55:52,461 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:55:52,461 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0337be0f939be2a2b593eefecee32758, store=C 2023-03-29 03:55:52,461 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:55:52,468 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=183 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/A/5e309c756055447bb1b9526249ab5159 2023-03-29 03:55:52,476 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=183 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/B/c288bc52f45740fba3a967ed962ba6a1 2023-03-29 03:55:52,486 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=183 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/C/c7d9d065f5924811bbd65187b9c47eec 2023-03-29 03:55:52,489 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/A/5e309c756055447bb1b9526249ab5159 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/5e309c756055447bb1b9526249ab5159 2023-03-29 03:55:52,493 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/5e309c756055447bb1b9526249ab5159, entries=150, sequenceid=183, filesize=11.8 K 2023-03-29 03:55:52,494 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/B/c288bc52f45740fba3a967ed962ba6a1 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/c288bc52f45740fba3a967ed962ba6a1 2023-03-29 03:55:52,499 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/c288bc52f45740fba3a967ed962ba6a1, entries=100, sequenceid=183, filesize=9.4 K 2023-03-29 03:55:52,499 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/C/c7d9d065f5924811bbd65187b9c47eec as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/c7d9d065f5924811bbd65187b9c47eec 2023-03-29 03:55:52,503 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/c7d9d065f5924811bbd65187b9c47eec, entries=100, sequenceid=183, filesize=9.4 K 2023-03-29 03:55:52,504 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=107.34 KB/109920 for 0337be0f939be2a2b593eefecee32758 in 44ms, sequenceid=183, compaction requested=true 2023-03-29 03:55:52,504 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 0337be0f939be2a2b593eefecee32758: 2023-03-29 03:55:52,504 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:55:52,504 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:55:52,504 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-29 03:55:52,504 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-29 03:55:52,504 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:55:52,505 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36514 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-29 03:55:52,505 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1912): 0337be0f939be2a2b593eefecee32758/A is initiating minor compaction (all files) 2023-03-29 03:55:52,505 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34124 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-29 03:55:52,505 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 0337be0f939be2a2b593eefecee32758/A in TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:52,505 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1912): 0337be0f939be2a2b593eefecee32758/B is initiating minor compaction (all files) 2023-03-29 03:55:52,505 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0337be0f939be2a2b593eefecee32758/B in TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:52,505 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/462210eb463f48a793a5da760b1f06c4, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/7b242d42d4e6411b9bb939897c7fb883, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/5e309c756055447bb1b9526249ab5159] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp, totalSize=35.7 K 2023-03-29 03:55:52,505 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/29eb1917b4d841779e0f57359d1101f0, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/5b0d803ed6f54678bb227f66581a9faf, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/c288bc52f45740fba3a967ed962ba6a1] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp, totalSize=33.3 K 2023-03-29 03:55:52,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on 0337be0f939be2a2b593eefecee32758 2023-03-29 03:55:52,506 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 0337be0f939be2a2b593eefecee32758 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-29 03:55:52,506 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0337be0f939be2a2b593eefecee32758, store=A 2023-03-29 03:55:52,506 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:55:52,507 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 29eb1917b4d841779e0f57359d1101f0, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=155, earliestPutTs=1680062150959 2023-03-29 03:55:52,507 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 462210eb463f48a793a5da760b1f06c4, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=155, earliestPutTs=1680062150959 2023-03-29 03:55:52,507 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0337be0f939be2a2b593eefecee32758, store=B 2023-03-29 03:55:52,507 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:55:52,507 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 7b242d42d4e6411b9bb939897c7fb883, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=173, earliestPutTs=1680062151316 2023-03-29 03:55:52,507 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0337be0f939be2a2b593eefecee32758, store=C 2023-03-29 03:55:52,507 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 5b0d803ed6f54678bb227f66581a9faf, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=173, earliestPutTs=1680062151316 2023-03-29 03:55:52,508 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:55:52,508 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 5e309c756055447bb1b9526249ab5159, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=183, earliestPutTs=1680062152454 2023-03-29 03:55:52,508 DEBUG [Thread-3261] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:52,508 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting c288bc52f45740fba3a967ed962ba6a1, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=183, earliestPutTs=1680062152454 2023-03-29 03:55:52,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:52,510 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:52,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:52,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:52,511 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:52,511 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:52,511 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:52,512 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:52,512 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:52,512 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:52,513 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:52,514 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:52,514 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:52,514 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:52,514 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:52,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:52,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:52,515 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:52,515 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0337be0f939be2a2b593eefecee32758#A#compaction#473 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:55:52,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:52,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:52,515 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:52,515 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:52,515 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:52,515 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:52,515 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:52,515 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:52,515 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:52,516 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:52,516 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:52,516 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. started... 2023-03-29 03:55:52,516 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. as already flushing 2023-03-29 03:55:52,516 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:52,516 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:52,516 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:52,516 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:52,516 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:52,516 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:52,518 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:52,519 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:52,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:52,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:52,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:52,519 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:52,519 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:52,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:52,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:52,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:52,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:52,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:52,521 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:52,521 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b3f67bc[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:52,521 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:52,521 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:52,522 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:52,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:52,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:52,522 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:52,522 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:52,522 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:52,522 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:52,523 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:52,524 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:52,524 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:52,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:52,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:52,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:52,524 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:52,524 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:52,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:52,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:52,525 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] throttle.PressureAwareThroughputController(145): 0337be0f939be2a2b593eefecee32758#B#compaction#474 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:55:52,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:52,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:52,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:52,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:52,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:52,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:52,527 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:52,527 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:52,527 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:52,527 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:52,527 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:52,527 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:52,529 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:52,529 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:52,529 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:52,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:52,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:52,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:52,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:52,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:52,529 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:52,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:52,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:52,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:52,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:52,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:52,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:52,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:52,531 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:52,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:52,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:52,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:52,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:52,536 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:52,536 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:52,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:52,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:52,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:52,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:52,540 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:52,540 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:52,540 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:52,540 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:52,540 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:52,540 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:52,540 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:52,540 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:55:52,540 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:52,540 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:52,540 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:52,540 DEBUG [Thread-3261] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:52,540 DEBUG [Thread-3261] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:52,541 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:52,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42004 deadline: 1680062212538, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:52,544 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/A/d1ae589112414c0094e6c2ec76a39979 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/d1ae589112414c0094e6c2ec76a39979 2023-03-29 03:55:52,546 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:52,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42010 deadline: 1680062212540, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:52,550 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/B/ad4dd405c3714039abc133505b1c57dc as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/ad4dd405c3714039abc133505b1c57dc 2023-03-29 03:55:52,550 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0337be0f939be2a2b593eefecee32758/A of 0337be0f939be2a2b593eefecee32758 into d1ae589112414c0094e6c2ec76a39979(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:55:52,550 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0337be0f939be2a2b593eefecee32758: 2023-03-29 03:55:52,550 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758., storeName=0337be0f939be2a2b593eefecee32758/A, priority=13, startTime=1680062152504; duration=0sec 2023-03-29 03:55:52,550 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:55:52,550 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-29 03:55:52,551 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34124 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-29 03:55:52,551 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1912): 0337be0f939be2a2b593eefecee32758/C is initiating minor compaction (all files) 2023-03-29 03:55:52,551 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 0337be0f939be2a2b593eefecee32758/C in TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:52,551 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/bf79062a472342099e8940b53d90c888, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/fb54b30d187644b9abcd14be0da33059, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/c7d9d065f5924811bbd65187b9c47eec] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp, totalSize=33.3 K 2023-03-29 03:55:52,552 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting bf79062a472342099e8940b53d90c888, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=155, earliestPutTs=1680062150959 2023-03-29 03:55:52,552 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting fb54b30d187644b9abcd14be0da33059, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=173, earliestPutTs=1680062151316 2023-03-29 03:55:52,552 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting c7d9d065f5924811bbd65187b9c47eec, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=183, earliestPutTs=1680062152454 2023-03-29 03:55:52,555 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0337be0f939be2a2b593eefecee32758/B of 0337be0f939be2a2b593eefecee32758 into ad4dd405c3714039abc133505b1c57dc(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:55:52,555 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0337be0f939be2a2b593eefecee32758: 2023-03-29 03:55:52,555 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758., storeName=0337be0f939be2a2b593eefecee32758/B, priority=13, startTime=1680062152504; duration=0sec 2023-03-29 03:55:52,555 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:55:52,558 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0337be0f939be2a2b593eefecee32758#C#compaction#475 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:55:52,564 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/C/dcf0e1b3ab894c2b9078c5157207ebf3 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/dcf0e1b3ab894c2b9078c5157207ebf3 2023-03-29 03:55:52,567 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0337be0f939be2a2b593eefecee32758/C of 0337be0f939be2a2b593eefecee32758 into dcf0e1b3ab894c2b9078c5157207ebf3(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:55:52,567 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0337be0f939be2a2b593eefecee32758: 2023-03-29 03:55:52,567 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758., storeName=0337be0f939be2a2b593eefecee32758/C, priority=13, startTime=1680062152504; duration=0sec 2023-03-29 03:55:52,567 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:55:52,641 DEBUG [Thread-3261] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:52,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:52,643 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:52,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:52,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:52,644 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:52,644 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:52,644 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:52,644 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:52,644 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:52,644 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:52,645 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:52,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42004 deadline: 1680062212642, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:52,645 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:52,645 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:52,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:52,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:52,645 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:52,645 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:52,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:52,646 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:52,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:52,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:52,646 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:52,646 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:52,646 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:52,646 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:52,647 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:52,647 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:52,647 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:52,647 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:52,647 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:52,647 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. started... 2023-03-29 03:55:52,647 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. as already flushing 2023-03-29 03:55:52,647 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:52,647 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:52,647 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:52,647 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:52,647 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:52,647 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:52,648 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:52,648 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:52,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:52,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:52,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:52,649 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:52,649 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:52,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:52,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:52,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:52,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:52,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:52,650 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:52,650 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2edda7b1[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:52,650 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:52,650 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:52,650 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:52,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42010 deadline: 1680062212647, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:52,651 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:52,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:52,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:52,651 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:52,651 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:52,652 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:52,652 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:52,652 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:52,654 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:52,654 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:52,654 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:52,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:52,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:52,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:52,654 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:52,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:52,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:52,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:52,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:52,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:52,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:52,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:52,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:52,657 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:52,657 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:52,657 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:52,657 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:52,657 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:52,657 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:52,658 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:52,658 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:52,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:52,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:52,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:52,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:52,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:52,658 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:52,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:52,659 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:52,659 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:52,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:52,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:52,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:52,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:52,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:52,660 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:52,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:52,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:52,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:52,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:52,663 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:52,663 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:52,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:52,663 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:52,663 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:52,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:52,663 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:52,663 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:52,663 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:52,663 DEBUG [Thread-3261] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:52,663 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:52,663 DEBUG [Thread-3261] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:52,663 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:52,663 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:52,663 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:52,663 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:52,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:52,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:52,763 DEBUG [Thread-3261] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:52,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:52,765 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:52,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:52,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:52,766 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:52,766 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:52,766 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:52,766 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:52,767 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:52,767 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:52,768 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:52,768 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:52,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:52,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:52,768 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:52,768 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:52,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:52,769 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:52,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:52,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:52,769 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:52,769 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:52,769 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:52,769 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:52,769 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:52,770 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:52,770 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:52,770 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:52,770 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:52,770 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. started... 2023-03-29 03:55:52,770 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. as already flushing 2023-03-29 03:55:52,770 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:52,770 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:52,770 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:52,770 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:52,770 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:52,770 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:52,772 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:52,772 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:52,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:52,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:52,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:52,773 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:52,773 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:52,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:52,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:52,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:52,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:52,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:52,776 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:52,776 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1ffadc75[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:52,776 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:52,776 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:52,777 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:52,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:52,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:52,777 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:52,777 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:52,777 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:52,777 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:52,777 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:52,778 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:52,778 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:52,778 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:52,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:52,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:52,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:52,778 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:52,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:52,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:52,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:52,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:52,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:52,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:52,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:52,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:52,782 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:52,782 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:52,782 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:52,782 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:52,782 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:52,782 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:52,784 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:52,784 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:52,784 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:52,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:52,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:52,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:52,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:52,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:52,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:52,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:52,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:52,784 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:52,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:52,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:52,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:52,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:52,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:52,786 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:52,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:52,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:52,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:52,789 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:52,789 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:52,789 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:52,789 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:52,789 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:52,790 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:52,790 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:52,790 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:52,790 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:52,790 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:52,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:52,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:52,790 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:52,790 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:52,790 DEBUG [Thread-3261] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:52,790 DEBUG [Thread-3261] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:52,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:52,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:52,849 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:52,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42004 deadline: 1680062212847, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:52,857 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:52,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42010 deadline: 1680062212852, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:52,890 DEBUG [Thread-3261] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:52,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:52,893 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:52,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:52,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:52,894 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:52,894 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:52,894 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:52,894 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:52,894 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:52,894 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:52,896 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:52,896 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:52,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:52,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:52,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:52,896 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:52,896 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:52,897 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:52,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:52,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:52,897 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:52,897 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:52,897 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:52,897 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:52,898 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:52,898 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:52,898 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:52,898 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:52,898 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:52,898 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. started... 2023-03-29 03:55:52,898 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. as already flushing 2023-03-29 03:55:52,898 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:52,898 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:52,898 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:52,898 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:52,898 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:52,898 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:52,900 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:52,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:52,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:52,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:52,900 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:52,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:52,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:52,901 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:52,901 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:52,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:52,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:52,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:52,902 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:52,902 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:52,902 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c1e5ae4[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:52,902 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:52,906 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:52,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:52,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:52,906 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:52,907 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:52,907 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:52,907 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:52,907 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:52,908 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:52,909 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:52,909 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:52,909 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:52,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:52,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:52,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:52,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:52,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:52,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:52,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:52,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:52,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:52,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:52,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:52,911 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:52,911 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:52,911 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:52,911 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:52,911 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:52,912 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:52,913 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:52,913 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:52,913 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:52,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:52,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:52,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:52,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:52,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:52,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:52,913 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:52,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:52,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:52,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:52,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:52,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:52,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:52,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:52,915 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:52,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:52,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:52,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:52,920 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:52,920 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:52,920 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:52,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:52,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:52,920 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:52,920 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:52,920 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:52,920 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:52,920 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:52,920 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:52,920 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:52,920 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:52,920 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:52,920 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:55:52,920 DEBUG [Thread-3261] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:52,920 DEBUG [Thread-3261] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:52,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:52,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:52,934 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=203 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/A/e1129ad98da248ba8a9cc4aca7797dc0 2023-03-29 03:55:52,946 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=203 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/B/ea3489642ac446068bb0ec3a0b7dac96 2023-03-29 03:55:53,021 DEBUG [Thread-3261] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:53,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:53,023 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:53,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:53,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:53,026 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:53,026 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:53,026 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:53,026 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:53,026 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:53,026 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:53,028 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:53,028 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:53,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:53,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:53,028 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:53,028 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:53,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:53,029 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:53,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:53,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:53,030 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:53,030 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:53,030 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:53,030 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:53,030 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:53,030 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:53,032 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:53,032 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:53,032 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:53,032 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. started... 2023-03-29 03:55:53,032 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. as already flushing 2023-03-29 03:55:53,032 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:53,032 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:53,032 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:53,032 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:53,032 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:53,032 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:53,034 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:53,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:53,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:53,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:53,034 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:53,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:53,036 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:53,036 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:53,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:53,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:53,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:53,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:53,037 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:53,038 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@cbcca5c[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:53,038 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:53,038 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:53,040 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:53,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:53,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:53,040 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:53,040 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:53,040 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:53,040 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:53,040 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:53,042 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:53,042 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:53,042 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:53,042 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:53,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:53,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:53,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:53,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:53,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:53,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:53,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:53,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:53,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:53,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:53,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:53,049 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:53,049 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:53,049 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:53,049 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:53,049 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:53,049 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:53,051 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:53,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:53,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:53,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:53,051 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:53,051 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:53,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:53,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:53,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:53,052 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:53,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:53,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:53,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:53,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:53,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:53,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:53,055 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:53,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:53,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:53,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:53,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:53,058 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:53,058 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:53,058 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:53,058 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:53,058 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:53,058 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:55:53,058 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:53,058 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:53,058 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:53,058 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:53,058 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:53,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:53,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:53,058 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:53,058 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:53,058 DEBUG [Thread-3261] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:53,059 DEBUG [Thread-3261] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:53,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:53,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:53,155 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:53,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42004 deadline: 1680062213153, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:53,159 DEBUG [Thread-3261] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:53,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:53,160 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:53,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:53,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:53,161 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:53,161 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:53,161 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:53,161 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:53,162 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:53,162 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:53,163 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:53,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42010 deadline: 1680062213159, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:53,164 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:53,164 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:53,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:53,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:53,164 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:53,164 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:53,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:53,165 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:53,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:53,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:53,165 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:53,165 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:53,165 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:53,165 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:53,165 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:53,165 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:53,165 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:53,165 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:53,165 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:53,165 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. started... 2023-03-29 03:55:53,166 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. as already flushing 2023-03-29 03:55:53,166 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:53,166 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:53,166 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:53,166 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:53,166 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:53,166 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:53,167 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:53,167 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:53,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:53,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:53,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:53,167 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:53,167 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:53,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:53,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:53,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:53,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:53,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:53,169 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:53,169 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f1f0455[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:53,169 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:53,169 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:53,170 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:53,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:53,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:53,170 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:53,170 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:53,170 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:53,170 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:53,170 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:53,171 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:53,171 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:53,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:53,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:53,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:53,171 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:53,171 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:53,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:53,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:53,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:53,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:53,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:53,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:53,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:53,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:53,173 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:53,173 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:53,173 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:53,173 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:53,173 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:53,173 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:53,176 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:53,176 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:53,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:53,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:53,176 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:53,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:53,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:53,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:53,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:53,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:53,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:53,177 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:53,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:53,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:53,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:53,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:53,178 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:53,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:53,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:53,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:53,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:53,180 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:53,180 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:53,180 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:53,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:53,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:53,180 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:53,181 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:53,180 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:53,180 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:53,180 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:53,181 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:53,181 DEBUG [Thread-3261] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:53,181 DEBUG [Thread-3261] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:53,181 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:53,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:53,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:53,181 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:53,181 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:53,281 DEBUG [Thread-3261] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:53,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:53,283 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:53,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:53,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:53,284 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:53,284 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:53,284 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:53,284 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:53,284 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:53,284 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:53,285 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:53,285 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:53,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:53,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:53,286 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:53,286 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:53,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:53,286 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:53,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:53,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:53,287 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:53,287 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:53,287 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:53,287 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:53,287 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:53,287 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:53,287 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:53,287 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:53,287 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:53,287 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. started... 2023-03-29 03:55:53,287 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. as already flushing 2023-03-29 03:55:53,287 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:53,288 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:53,288 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:53,288 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:53,288 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:53,288 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:53,289 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:53,289 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:53,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:53,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:53,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:53,289 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:53,289 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:53,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:53,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:53,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:53,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:53,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:53,291 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:53,291 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@64f60b7[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:53,291 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:53,291 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:53,292 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:53,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:53,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:53,293 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:53,293 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:53,293 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:53,293 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:53,293 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:53,294 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:53,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:53,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:53,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:53,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:53,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:53,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:53,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:53,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:53,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:53,295 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:53,295 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:53,296 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:53,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:53,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:53,296 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:53,296 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:53,296 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:53,296 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:53,296 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:53,296 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:53,298 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:53,298 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:53,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:53,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:53,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:53,298 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:53,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:53,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:53,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:53,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:53,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:53,298 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:53,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:53,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:53,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:53,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:53,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:53,299 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:53,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:53,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:53,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:53,302 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:53,302 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:53,302 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:53,302 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:53,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:53,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:53,302 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:53,302 DEBUG [Thread-3261] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:53,302 DEBUG [Thread-3261] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:53,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:53,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:53,303 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:53,303 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:53,303 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:53,303 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:53,303 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:53,303 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:53,303 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:53,355 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=203 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/C/30b14896f056491e8afcc3c6d49197ab 2023-03-29 03:55:53,359 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/A/e1129ad98da248ba8a9cc4aca7797dc0 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/e1129ad98da248ba8a9cc4aca7797dc0 2023-03-29 03:55:53,362 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/e1129ad98da248ba8a9cc4aca7797dc0, entries=200, sequenceid=203, filesize=14.1 K 2023-03-29 03:55:53,363 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/B/ea3489642ac446068bb0ec3a0b7dac96 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/ea3489642ac446068bb0ec3a0b7dac96 2023-03-29 03:55:53,366 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/ea3489642ac446068bb0ec3a0b7dac96, entries=150, sequenceid=203, filesize=11.8 K 2023-03-29 03:55:53,366 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/C/30b14896f056491e8afcc3c6d49197ab as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/30b14896f056491e8afcc3c6d49197ab 2023-03-29 03:55:53,369 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/30b14896f056491e8afcc3c6d49197ab, entries=150, sequenceid=203, filesize=11.8 K 2023-03-29 03:55:53,370 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 0337be0f939be2a2b593eefecee32758 in 864ms, sequenceid=203, compaction requested=false 2023-03-29 03:55:53,370 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 0337be0f939be2a2b593eefecee32758: 2023-03-29 03:55:53,402 DEBUG [Thread-3261] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:53,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:53,404 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:53,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:53,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:53,405 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:53,405 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:53,405 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:53,405 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:53,405 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:53,405 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:53,407 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:53,407 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:53,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:53,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:53,408 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:53,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:53,408 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:53,408 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:53,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:53,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:53,408 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:53,409 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:53,409 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:53,409 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:53,409 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:53,409 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:53,409 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:53,409 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:53,409 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:53,409 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. started... 2023-03-29 03:55:53,409 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0337be0f939be2a2b593eefecee32758 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-29 03:55:53,409 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0337be0f939be2a2b593eefecee32758, store=A 2023-03-29 03:55:53,409 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:55:53,410 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0337be0f939be2a2b593eefecee32758, store=B 2023-03-29 03:55:53,410 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:55:53,410 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0337be0f939be2a2b593eefecee32758, store=C 2023-03-29 03:55:53,410 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:55:53,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on 0337be0f939be2a2b593eefecee32758 2023-03-29 03:55:53,662 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. as already flushing 2023-03-29 03:55:53,718 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:53,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42010 deadline: 1680062213713, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:53,718 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:53,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42004 deadline: 1680062213714, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:53,818 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=218 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/A/fbb43625dbe84b1c9c268879845860b8 2023-03-29 03:55:53,822 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:53,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42010 deadline: 1680062213819, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:53,822 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:53,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42004 deadline: 1680062213819, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:54,026 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:54,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42010 deadline: 1680062214023, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:54,026 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:54,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42004 deadline: 1680062214023, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:54,227 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=218 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/B/02cb896a46b44d96862d932e28c317bd 2023-03-29 03:55:54,242 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=218 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/C/e18a0008c2ed428babfa74b8c2f411e9 2023-03-29 03:55:54,246 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/A/fbb43625dbe84b1c9c268879845860b8 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/fbb43625dbe84b1c9c268879845860b8 2023-03-29 03:55:54,256 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/fbb43625dbe84b1c9c268879845860b8, entries=150, sequenceid=218, filesize=11.8 K 2023-03-29 03:55:54,257 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/B/02cb896a46b44d96862d932e28c317bd as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/02cb896a46b44d96862d932e28c317bd 2023-03-29 03:55:54,261 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/02cb896a46b44d96862d932e28c317bd, entries=150, sequenceid=218, filesize=11.8 K 2023-03-29 03:55:54,263 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/C/e18a0008c2ed428babfa74b8c2f411e9 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/e18a0008c2ed428babfa74b8c2f411e9 2023-03-29 03:55:54,266 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/e18a0008c2ed428babfa74b8c2f411e9, entries=150, sequenceid=218, filesize=11.8 K 2023-03-29 03:55:54,267 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 0337be0f939be2a2b593eefecee32758 in 858ms, sequenceid=218, compaction requested=true 2023-03-29 03:55:54,267 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0337be0f939be2a2b593eefecee32758: 2023-03-29 03:55:54,267 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:54,267 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:54,267 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:54,267 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:54,267 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:54,267 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:54,271 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:54,272 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:54,272 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:54,272 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:54,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:54,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:54,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:54,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:54,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:54,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:54,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:54,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:54,274 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:54,275 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@58d8d3e0[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:54,275 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:54,275 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:54,280 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:54,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:54,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:54,280 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:54,280 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:54,280 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:54,280 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:54,280 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:54,281 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:54,282 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:54,282 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:54,283 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:54,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:54,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:54,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:54,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:54,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:54,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:54,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:54,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:54,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:54,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:54,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:54,285 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:54,285 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:54,285 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:54,285 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:54,285 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:54,286 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:54,287 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:54,287 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:54,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:54,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:54,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:54,287 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:54,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:54,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:54,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:54,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:54,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:54,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:54,288 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:54,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:54,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:54,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:54,289 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:54,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:54,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:54,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:54,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:54,293 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:54,293 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:54,293 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:54,293 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:54,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:54,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:54,293 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:54,293 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:54,293 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:54,293 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:54,293 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:54,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:54,293 DEBUG [Thread-3261] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:54,293 DEBUG [Thread-3261] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:54,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:54,293 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:54,293 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:54,293 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:54,333 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680062153667, max timestamp=1680062154331Num uniques 859; 2023-03-29 03:55:54,333 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680062153667, max timestamp=1680062154331Num uniques 859; 2023-03-29 03:55:54,333 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680062153667, max timestamp=1680062154331Num uniques 859; 2023-03-29 03:55:54,333 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-29 03:55:54,333 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-29 03:55:54,333 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-29 03:55:54,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on 0337be0f939be2a2b593eefecee32758 2023-03-29 03:55:54,345 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 0337be0f939be2a2b593eefecee32758 3/3 column families, dataSize=53.67 KB heapSize=145.92 KB 2023-03-29 03:55:54,345 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0337be0f939be2a2b593eefecee32758, store=A 2023-03-29 03:55:54,345 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-29 03:55:54,345 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0337be0f939be2a2b593eefecee32758, store=B 2023-03-29 03:55:54,345 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-29 03:55:54,345 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0337be0f939be2a2b593eefecee32758, store=C 2023-03-29 03:55:54,345 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-29 03:55:54,393 DEBUG [Thread-3261] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:54,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:54,396 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:54,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:54,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:54,397 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:54,397 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:54,397 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:54,397 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:54,397 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:54,397 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:54,400 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:54,400 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:54,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:54,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:54,400 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:54,400 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:54,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:54,401 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:54,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:54,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:54,401 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:54,401 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:54,401 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:54,401 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:54,401 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:54,402 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:54,409 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:54,409 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:54,409 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:54,410 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. started... 2023-03-29 03:55:54,410 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. as already flushing 2023-03-29 03:55:54,410 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:54,410 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:54,410 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:54,410 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:54,410 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:54,410 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:54,412 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:54,412 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:54,412 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:54,412 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:54,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:54,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:54,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:54,415 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680062154389, max timestamp=1680062154413Num uniques 859; 2023-03-29 03:55:54,415 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-29 03:55:54,415 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680062154389, max timestamp=1680062154413Num uniques 859; 2023-03-29 03:55:54,415 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-29 03:55:54,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:54,417 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680062154389, max timestamp=1680062154413Num uniques 859; 2023-03-29 03:55:54,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:54,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:54,417 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-29 03:55:54,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:54,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:54,418 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:54,418 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@32dc1c8d[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:54,418 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:54,418 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:54,419 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:54,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:54,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:54,419 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:54,419 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:54,419 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:54,419 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:54,419 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:54,421 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:54,421 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:54,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:54,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:54,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:54,421 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:54,421 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:54,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:54,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:54,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:54,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:54,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:54,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:54,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:54,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:54,423 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:54,423 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:54,423 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:54,423 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:54,423 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:54,423 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:54,431 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:54,431 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:54,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:54,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:54,431 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:54,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:54,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:54,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:54,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:54,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:54,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:54,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:54,432 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:54,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:54,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:54,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:54,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:54,434 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:54,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:54,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:54,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:54,437 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:54,437 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:54,438 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:54,438 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:54,438 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:54,438 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:54,438 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:54,438 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:54,438 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:54,438 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:54,438 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:54,438 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:54,438 DEBUG [Thread-3261] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:54,438 DEBUG [Thread-3261] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:54,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:54,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:54,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:54,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:54,485 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:54,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42004 deadline: 1680062214479, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:54,499 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:54,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42010 deadline: 1680062214483, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:54,500 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:54,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42024 deadline: 1680062214483, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:54,500 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:54,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41942 deadline: 1680062214485, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:54,505 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:54,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41960 deadline: 1680062214497, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:54,511 DEBUG [Thread-3251] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8208 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758., hostname=jenkins-hbase4.apache.org,35599,1680062018534, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-29 03:55:54,538 DEBUG [Thread-3261] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:54,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:54,540 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:54,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:54,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:54,545 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:54,545 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:54,545 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:54,545 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:54,545 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:54,545 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:54,547 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:54,547 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:54,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:54,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:54,547 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:54,547 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:54,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:54,548 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:54,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:54,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:54,548 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:54,548 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:54,548 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:54,548 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:54,548 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:54,548 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:54,551 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:54,551 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:54,551 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:54,551 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. started... 2023-03-29 03:55:54,551 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. as already flushing 2023-03-29 03:55:54,551 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:54,551 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:54,551 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:54,551 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:54,551 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:54,551 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:54,553 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:54,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:54,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:54,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:54,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:54,553 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:54,560 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:54,560 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:54,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:54,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:54,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:54,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:54,567 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:54,568 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5569b2f[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:54,568 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:54,568 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:54,570 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:54,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:54,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:54,570 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:54,570 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:54,570 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:54,570 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:54,570 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:54,572 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:54,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:54,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:54,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:54,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:54,572 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:54,572 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:54,572 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:54,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:54,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:54,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:54,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:54,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:54,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:54,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:54,574 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:54,574 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:54,574 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:54,574 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:54,574 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:54,574 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:54,577 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:54,577 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:54,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:54,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:54,577 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:54,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:54,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:54,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:54,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:54,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:54,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:54,578 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:54,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:54,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:54,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:54,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:54,580 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:54,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:54,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:54,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:54,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:54,583 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:54,583 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:54,583 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:54,583 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:54,583 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:54,583 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:54,583 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:54,583 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:54,583 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:54,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:54,583 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:54,583 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:54,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:54,583 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:54,583 DEBUG [Thread-3261] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:54,583 DEBUG [Thread-3261] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:54,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:54,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:54,584 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:55:54,595 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:54,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42004 deadline: 1680062214586, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:54,613 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:54,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42024 deadline: 1680062214602, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:54,613 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:54,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41942 deadline: 1680062214602, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:54,620 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:54,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42010 deadline: 1680062214612, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:54,684 DEBUG [Thread-3261] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:54,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:54,686 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:54,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:54,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:54,687 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:54,687 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:54,687 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:54,687 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:54,687 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:54,687 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:54,689 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:54,689 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:54,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:54,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:54,689 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:54,689 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:54,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:54,690 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:54,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:54,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:54,690 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:54,690 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:54,690 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:54,690 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:54,691 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:54,691 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:54,691 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:54,691 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:54,691 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:54,691 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. started... 2023-03-29 03:55:54,691 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. as already flushing 2023-03-29 03:55:54,691 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:54,691 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:54,691 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:54,691 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:54,691 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:54,691 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:54,692 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:54,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:54,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:54,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:54,693 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:54,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:54,693 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:54,693 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:54,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:54,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:54,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:54,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:54,694 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:54,694 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@786109ea[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:54,694 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:54,694 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:54,695 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:54,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:54,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:54,695 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:54,695 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:54,695 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:54,695 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:54,695 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:54,698 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:54,698 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:54,698 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:54,698 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:54,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:54,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:54,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:54,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:54,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:54,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:54,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:54,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:54,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:54,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:54,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:54,700 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:54,700 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:54,700 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:54,700 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:54,700 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:54,700 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:54,701 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:54,702 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:54,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:54,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:54,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:54,702 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:54,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:54,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:54,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:54,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:54,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:54,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:54,703 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:54,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:54,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:54,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:54,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:54,704 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:54,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:54,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:54,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:54,706 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:54,706 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:54,706 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:54,707 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:54,707 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:54,707 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:54,707 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:54,707 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:54,707 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:54,707 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:54,707 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:54,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:54,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:54,707 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:54,707 DEBUG [Thread-3261] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:54,707 DEBUG [Thread-3261] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:54,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:54,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:54,781 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=243 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/A/0a726acce5094c06a95800bd6276db19 2023-03-29 03:55:54,801 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:54,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42004 deadline: 1680062214798, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:54,807 DEBUG [Thread-3261] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:54,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:54,809 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:54,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:54,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:54,810 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:54,810 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:54,810 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:54,810 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:54,811 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:54,811 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:54,812 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:54,812 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:54,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:54,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:54,812 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:54,812 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:54,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:54,812 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:54,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:54,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:54,813 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:54,813 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:54,813 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:54,813 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:54,813 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:54,813 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:54,813 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:54,813 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:54,813 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:54,813 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. started... 2023-03-29 03:55:54,813 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. as already flushing 2023-03-29 03:55:54,813 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:54,814 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:54,814 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:54,814 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:54,814 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:54,814 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:54,816 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:54,816 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:54,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:54,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:54,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:54,817 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:54,817 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:54,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:54,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:54,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:54,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:54,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:54,818 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:54,818 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@66b892d5[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:54,818 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:54,818 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:54,819 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:54,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:54,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:54,819 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:54,819 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:54,819 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:54,820 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:54,819 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:54,821 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:54,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42024 deadline: 1680062214814, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:54,821 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:54,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41942 deadline: 1680062214816, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:54,821 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:54,821 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:54,821 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:54,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:54,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:54,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:54,821 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:54,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:54,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:54,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:54,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:54,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:54,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:54,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:54,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:54,823 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:54,823 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:54,823 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:54,823 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:54,823 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:54,824 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:54,825 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:54,825 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:54,825 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:54,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:54,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:54,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:54,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:54,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:54,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:54,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:54,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:54,826 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:54,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42010 deadline: 1680062214822, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:54,826 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:54,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:54,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:54,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:54,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:54,827 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:54,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:54,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:54,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:54,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:54,830 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:54,830 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:54,830 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:54,830 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:54,830 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:54,830 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:54,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:54,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:54,830 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:54,830 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:54,830 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:54,830 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:54,830 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:54,830 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:54,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:54,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:54,830 DEBUG [Thread-3261] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:54,830 DEBUG [Thread-3261] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:54,930 DEBUG [Thread-3261] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:54,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:54,932 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:54,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:54,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:54,933 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:54,933 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:54,933 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:54,933 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:54,934 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:54,934 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:54,935 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:54,935 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:54,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:54,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:54,935 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:54,935 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:54,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:54,936 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:54,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:54,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:54,936 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:54,936 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:54,936 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:54,936 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:54,937 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:54,937 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:54,937 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:54,937 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:54,937 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:54,937 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. started... 2023-03-29 03:55:54,937 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. as already flushing 2023-03-29 03:55:54,937 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:54,937 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:54,937 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:54,937 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:54,937 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:54,937 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:54,938 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:54,939 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:54,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:54,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:54,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:54,939 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:54,939 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:54,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:54,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:54,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:54,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:54,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:54,941 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:54,941 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@34d2c2fb[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:54,941 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:54,941 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:54,942 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:54,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:54,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:54,942 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:54,942 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:54,942 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:54,942 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:54,942 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:54,944 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:54,944 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:54,944 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:54,944 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:54,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:54,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:54,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:54,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:54,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:54,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:54,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:54,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:54,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:54,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:54,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:54,947 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:54,947 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:54,947 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:54,947 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:54,947 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:54,947 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:54,948 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:54,948 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:54,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:54,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:54,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:54,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:54,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:54,948 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:54,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:54,949 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:54,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:54,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:54,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:54,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:54,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:54,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:54,950 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:54,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:54,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:54,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:54,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:54,952 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:54,952 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:54,952 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:54,952 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:54,952 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:54,952 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:54,953 DEBUG [Thread-3261] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:54,953 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:54,953 DEBUG [Thread-3261] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:54,953 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:54,953 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:54,953 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:54,956 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:54,956 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:54,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:54,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:54,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:54,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:55,053 DEBUG [Thread-3261] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:55,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:55,061 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:55,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:55,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:55,063 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:55,063 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:55,063 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:55,063 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:55,064 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:55,064 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:55,065 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:55,065 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:55,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:55,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:55,066 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:55,066 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:55,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:55,066 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:55,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:55,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:55,067 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:55,067 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:55,067 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:55,067 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:55,067 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:55,067 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:55,068 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:55,068 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:55,068 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:55,068 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. started... 2023-03-29 03:55:55,068 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. as already flushing 2023-03-29 03:55:55,068 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:55,068 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:55,068 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:55,068 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:55,068 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:55,068 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:55,070 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:55,070 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:55,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:55,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:55,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:55,070 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:55,070 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:55,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:55,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:55,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:55,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:55,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:55,071 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:55,072 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ecb8ad[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:55,072 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:55,072 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:55,074 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:55,074 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:55,074 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:55,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:55,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:55,074 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:55,074 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:55,074 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:55,076 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:55,076 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:55,076 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:55,076 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:55,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:55,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:55,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:55,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:55,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:55,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:55,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:55,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:55,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:55,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:55,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:55,078 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:55,078 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:55,078 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:55,078 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:55,078 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:55,078 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:55,082 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:55,083 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:55,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:55,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:55,083 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:55,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:55,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:55,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:55,088 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:55,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:55,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:55,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:55,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:55,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:55,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:55,089 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:55,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:55,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:55,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:55,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:55,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:55,092 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:55,093 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:55,093 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:55,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:55,093 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:55,093 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:55,093 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:55,093 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:55,093 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:55,093 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:55,093 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:55,093 DEBUG [Thread-3261] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:55,093 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:55,093 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:55:55,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:55,093 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:55,093 DEBUG [Thread-3261] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:55,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:55,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:55,110 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:55,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42004 deadline: 1680062215103, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:55,127 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:55,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42024 deadline: 1680062215124, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:55,127 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:55,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41942 deadline: 1680062215124, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:55,133 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:55,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42010 deadline: 1680062215128, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:55,191 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=243 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/B/585023d84be347cfae77d25f72e093a5 2023-03-29 03:55:55,193 DEBUG [Thread-3261] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:55,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:55,199 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:55,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:55,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:55,200 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:55,200 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:55,200 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:55,200 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:55,200 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:55,200 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:55,203 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:55,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:55,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:55,203 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:55,203 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=243 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/C/097531d3a75941288a3e28d3ff19e344 2023-03-29 03:55:55,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:55,203 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:55,203 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:55,204 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:55,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:55,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:55,204 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:55,205 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:55,205 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:55,205 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:55,205 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:55,205 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:55,209 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/A/0a726acce5094c06a95800bd6276db19 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/0a726acce5094c06a95800bd6276db19 2023-03-29 03:55:55,212 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:55,212 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:55,212 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:55,212 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. started... 2023-03-29 03:55:55,212 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. as already flushing 2023-03-29 03:55:55,212 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:55,212 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:55,212 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:55,212 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:55,212 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/0a726acce5094c06a95800bd6276db19, entries=250, sequenceid=243, filesize=16.4 K 2023-03-29 03:55:55,213 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:55,213 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:55,213 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/B/585023d84be347cfae77d25f72e093a5 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/585023d84be347cfae77d25f72e093a5 2023-03-29 03:55:55,214 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:55,214 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:55,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:55,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:55,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:55,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:55,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:55,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:55,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:55,216 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:55,216 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:55,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:55,217 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:55,217 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@75b2cfa3[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:55,217 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:55,217 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:55,218 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:55,218 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:55,218 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/585023d84be347cfae77d25f72e093a5, entries=150, sequenceid=243, filesize=11.8 K 2023-03-29 03:55:55,219 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/C/097531d3a75941288a3e28d3ff19e344 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/097531d3a75941288a3e28d3ff19e344 2023-03-29 03:55:55,220 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:55,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:55,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:55,220 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:55,220 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:55,220 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:55,222 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:55,222 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:55,223 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:55,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:55,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:55,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:55,223 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:55,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:55,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:55,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:55,224 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/097531d3a75941288a3e28d3ff19e344, entries=150, sequenceid=243, filesize=11.8 K 2023-03-29 03:55:55,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:55,224 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~145.88 KB/149376, currentSize=134.18 KB/137400 for 0337be0f939be2a2b593eefecee32758 in 879ms, sequenceid=243, compaction requested=true 2023-03-29 03:55:55,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:55,224 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 0337be0f939be2a2b593eefecee32758: 2023-03-29 03:55:55,225 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:55:55,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:55,225 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-29 03:55:55,225 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-29 03:55:55,225 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:55:55,227 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 55826 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-29 03:55:55,227 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1912): 0337be0f939be2a2b593eefecee32758/A is initiating minor compaction (all files) 2023-03-29 03:55:55,227 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 0337be0f939be2a2b593eefecee32758/A in TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:55,227 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/d1ae589112414c0094e6c2ec76a39979, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/e1129ad98da248ba8a9cc4aca7797dc0, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/fbb43625dbe84b1c9c268879845860b8, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/0a726acce5094c06a95800bd6276db19] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp, totalSize=54.5 K 2023-03-29 03:55:55,227 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48656 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-29 03:55:55,227 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1912): 0337be0f939be2a2b593eefecee32758/B is initiating minor compaction (all files) 2023-03-29 03:55:55,227 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0337be0f939be2a2b593eefecee32758/B in TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:55,227 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/ad4dd405c3714039abc133505b1c57dc, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/ea3489642ac446068bb0ec3a0b7dac96, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/02cb896a46b44d96862d932e28c317bd, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/585023d84be347cfae77d25f72e093a5] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp, totalSize=47.5 K 2023-03-29 03:55:55,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:55,228 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting d1ae589112414c0094e6c2ec76a39979, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=183, earliestPutTs=1680062151316 2023-03-29 03:55:55,228 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:55:55,228 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting ad4dd405c3714039abc133505b1c57dc, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=183, earliestPutTs=1680062151316 2023-03-29 03:55:55,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:55,228 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting e1129ad98da248ba8a9cc4aca7797dc0, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=203, earliestPutTs=1680062152484 2023-03-29 03:55:55,228 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:55,228 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:55,229 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:55,229 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:55,229 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:55,229 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting fbb43625dbe84b1c9c268879845860b8, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=218, earliestPutTs=1680062152517 2023-03-29 03:55:55,228 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:55,229 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting ea3489642ac446068bb0ec3a0b7dac96, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=203, earliestPutTs=1680062152484 2023-03-29 03:55:55,229 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 0a726acce5094c06a95800bd6276db19, keycount=250, bloomtype=ROW, size=16.4 K, encoding=NONE, compression=NONE, seqNum=243, earliestPutTs=1680062153713 2023-03-29 03:55:55,230 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 02cb896a46b44d96862d932e28c317bd, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=218, earliestPutTs=1680062152517 2023-03-29 03:55:55,231 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 585023d84be347cfae77d25f72e093a5, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=243, earliestPutTs=1680062153713 2023-03-29 03:55:55,232 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:55,232 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:55,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:55,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:55,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:55,232 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:55,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:55,232 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:55,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:55,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:55,233 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:55,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:55,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:55,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:55,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:55,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:55,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:55,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:55,236 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:55,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:55,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:55,240 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:55,240 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:55,240 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:55,240 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:55,241 DEBUG [Thread-3261] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:55,241 DEBUG [Thread-3261] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:55,240 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:55,241 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] throttle.PressureAwareThroughputController(145): 0337be0f939be2a2b593eefecee32758#B#compaction#484 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:55:55,242 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:55,242 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:55,242 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:55,242 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:55,242 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:55,242 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:55,242 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:55,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:55,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:55,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:55,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:55,246 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0337be0f939be2a2b593eefecee32758#A#compaction#485 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:55:55,260 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/B/bfabbcc60fdf4b76bf1701a2dd4b285d as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/bfabbcc60fdf4b76bf1701a2dd4b285d 2023-03-29 03:55:55,264 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 0337be0f939be2a2b593eefecee32758/B of 0337be0f939be2a2b593eefecee32758 into bfabbcc60fdf4b76bf1701a2dd4b285d(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:55:55,264 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0337be0f939be2a2b593eefecee32758: 2023-03-29 03:55:55,264 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758., storeName=0337be0f939be2a2b593eefecee32758/B, priority=12, startTime=1680062155225; duration=0sec 2023-03-29 03:55:55,264 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:55:55,264 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-29 03:55:55,265 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48656 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-29 03:55:55,265 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1912): 0337be0f939be2a2b593eefecee32758/C is initiating minor compaction (all files) 2023-03-29 03:55:55,265 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0337be0f939be2a2b593eefecee32758/C in TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:55,265 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/dcf0e1b3ab894c2b9078c5157207ebf3, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/30b14896f056491e8afcc3c6d49197ab, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/e18a0008c2ed428babfa74b8c2f411e9, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/097531d3a75941288a3e28d3ff19e344] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp, totalSize=47.5 K 2023-03-29 03:55:55,266 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting dcf0e1b3ab894c2b9078c5157207ebf3, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=183, earliestPutTs=1680062151316 2023-03-29 03:55:55,266 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 30b14896f056491e8afcc3c6d49197ab, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=203, earliestPutTs=1680062152484 2023-03-29 03:55:55,266 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting e18a0008c2ed428babfa74b8c2f411e9, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=218, earliestPutTs=1680062152517 2023-03-29 03:55:55,266 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 097531d3a75941288a3e28d3ff19e344, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=243, earliestPutTs=1680062153713 2023-03-29 03:55:55,273 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] throttle.PressureAwareThroughputController(145): 0337be0f939be2a2b593eefecee32758#C#compaction#486 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:55:55,286 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/C/eb4a7aafeb7b4e068c8b7db465c15f54 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/eb4a7aafeb7b4e068c8b7db465c15f54 2023-03-29 03:55:55,289 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 0337be0f939be2a2b593eefecee32758/C of 0337be0f939be2a2b593eefecee32758 into eb4a7aafeb7b4e068c8b7db465c15f54(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:55:55,289 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0337be0f939be2a2b593eefecee32758: 2023-03-29 03:55:55,289 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758., storeName=0337be0f939be2a2b593eefecee32758/C, priority=12, startTime=1680062155228; duration=0sec 2023-03-29 03:55:55,289 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:55:55,341 DEBUG [Thread-3261] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:55,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:55,343 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:55,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:55,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:55,344 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:55,344 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:55,344 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:55,344 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:55,344 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:55,345 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:55,346 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:55,346 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:55,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:55,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:55,346 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:55,346 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:55,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:55,349 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:55,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:55,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:55,350 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:55,350 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:55,350 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:55,350 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:55,350 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:55,350 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:55,350 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:55,350 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:55,350 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:55,350 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. started... 2023-03-29 03:55:55,350 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0337be0f939be2a2b593eefecee32758 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-03-29 03:55:55,351 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0337be0f939be2a2b593eefecee32758, store=A 2023-03-29 03:55:55,351 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-29 03:55:55,351 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0337be0f939be2a2b593eefecee32758, store=B 2023-03-29 03:55:55,351 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-29 03:55:55,351 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0337be0f939be2a2b593eefecee32758, store=C 2023-03-29 03:55:55,351 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-29 03:55:55,372 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=281 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/A/9cb1db4d28db4c4884d5c646b4b2be8c 2023-03-29 03:55:55,390 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=281 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/B/02125a7e4e2844d59f622f66e4a647ed 2023-03-29 03:55:55,411 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=281 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/C/5b69893bd5314bc88f715dbae7065fae 2023-03-29 03:55:55,420 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/A/9cb1db4d28db4c4884d5c646b4b2be8c as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/9cb1db4d28db4c4884d5c646b4b2be8c 2023-03-29 03:55:55,426 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/9cb1db4d28db4c4884d5c646b4b2be8c, entries=150, sequenceid=281, filesize=11.9 K 2023-03-29 03:55:55,427 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/B/02125a7e4e2844d59f622f66e4a647ed as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/02125a7e4e2844d59f622f66e4a647ed 2023-03-29 03:55:55,431 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/02125a7e4e2844d59f622f66e4a647ed, entries=150, sequenceid=281, filesize=11.9 K 2023-03-29 03:55:55,431 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/C/5b69893bd5314bc88f715dbae7065fae as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/5b69893bd5314bc88f715dbae7065fae 2023-03-29 03:55:55,436 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/5b69893bd5314bc88f715dbae7065fae, entries=150, sequenceid=281, filesize=11.9 K 2023-03-29 03:55:55,436 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=0 B/0 for 0337be0f939be2a2b593eefecee32758 in 86ms, sequenceid=281, compaction requested=false 2023-03-29 03:55:55,436 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0337be0f939be2a2b593eefecee32758: 2023-03-29 03:55:55,436 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:55,436 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:55,436 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:55,436 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:55,440 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:55,440 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:55,448 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:55,448 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:55,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:55,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:55,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:55,448 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:55,448 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:55,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:55,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:55,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:55,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:55,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:55,450 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:55,450 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@57bed5fa[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:55,450 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:55,450 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:55,451 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:55,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:55,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:55,452 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:55,452 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:55,452 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:55,452 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:55,452 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:55,454 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:55,454 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:55,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:55,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:55,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:55,454 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:55,454 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:55,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:55,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:55,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:55,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:55,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:55,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:55,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:55,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:55,457 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:55,457 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:55,457 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:55,457 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:55,457 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:55,457 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:55,459 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:55,459 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:55,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:55,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:55,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:55,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:55,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:55,459 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:55,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:55,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:55,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:55,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:55,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:55,462 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:55,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:55,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:55,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:55,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:55,463 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:55,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:55,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:55,467 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:55,467 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:55,467 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:55,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:55,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:55,467 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:55,468 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:55,468 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:55,468 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:55,468 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:55,468 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:55,467 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:55,467 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:55,468 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:55,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:55,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:55,468 DEBUG [Thread-3261] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:55,468 DEBUG [Thread-3261] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:55,568 DEBUG [Thread-3261] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:55,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:55,571 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:55,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:55,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:55,572 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:55,572 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:55,572 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:55,572 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:55,573 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:55,573 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:55,574 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:55,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:55,574 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:55,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:55,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:55,574 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:55,574 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:55,575 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:55,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:55,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:55,575 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:55,575 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:55,575 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:55,575 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:55,575 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:55,575 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:55,575 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:55,575 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:55,575 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:55,576 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. started... 2023-03-29 03:55:55,576 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0337be0f939be2a2b593eefecee32758: 2023-03-29 03:55:55,576 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:55,576 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:55,576 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:55,576 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:55,576 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:55,576 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:55,582 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:55,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:55,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:55,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:55,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:55,583 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:55,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:55,583 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:55,583 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:55,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:55,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:55,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:55,584 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:55,584 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@755900ec[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:55,584 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:55,584 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:55,586 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:55,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:55,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:55,586 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:55,586 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:55,586 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:55,587 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:55,587 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:55,588 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:55,588 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:55,588 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:55,588 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:55,588 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:55,588 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:55,588 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:55,588 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:55,588 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:55,590 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:55,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:55,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:55,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:55,590 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:55,590 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:55,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:55,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:55,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:55,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:55,590 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:55,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:55,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:55,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:55,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:55,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:55,592 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:55,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:55,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:55,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:55,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:55,599 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:55,599 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:55,599 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:55,599 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:55,599 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:55,600 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:55,600 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:55,599 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:55,600 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:55,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:55,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:55,600 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:55,600 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:55:55,600 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:55,600 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:55,600 DEBUG [Thread-3261] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:55,600 DEBUG [Thread-3261] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:55,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:55,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:55,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on 0337be0f939be2a2b593eefecee32758 2023-03-29 03:55:55,635 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 0337be0f939be2a2b593eefecee32758 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-29 03:55:55,635 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0337be0f939be2a2b593eefecee32758, store=A 2023-03-29 03:55:55,635 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:55:55,635 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0337be0f939be2a2b593eefecee32758, store=B 2023-03-29 03:55:55,635 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:55:55,635 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0337be0f939be2a2b593eefecee32758, store=C 2023-03-29 03:55:55,635 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:55:55,645 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=292 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/A/0cc29faa019e4fef853c61dd1783e531 2023-03-29 03:55:55,659 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=292 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/B/6460c1d108414bbfab636286877c294e 2023-03-29 03:55:55,664 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/A/68f036894ad443fdb6aa41eb6bd4b127 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/68f036894ad443fdb6aa41eb6bd4b127 2023-03-29 03:55:55,668 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 0337be0f939be2a2b593eefecee32758/A of 0337be0f939be2a2b593eefecee32758 into 68f036894ad443fdb6aa41eb6bd4b127(size=12.4 K), total size for store is 24.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:55:55,668 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0337be0f939be2a2b593eefecee32758: 2023-03-29 03:55:55,668 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758., storeName=0337be0f939be2a2b593eefecee32758/A, priority=12, startTime=1680062155225; duration=0sec 2023-03-29 03:55:55,668 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:55:55,674 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11854, totalHeapSize=27176, min timestamp=1680062155657, max timestamp=1680062155672Num uniques 859; 2023-03-29 03:55:55,674 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=11854, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-29 03:55:55,674 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11854, totalHeapSize=27176, min timestamp=1680062155657, max timestamp=1680062155672Num uniques 859; 2023-03-29 03:55:55,674 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=11854, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-29 03:55:55,674 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11854, totalHeapSize=27176, min timestamp=1680062155657, max timestamp=1680062155672Num uniques 859; 2023-03-29 03:55:55,674 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=11854, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-29 03:55:55,676 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=292 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/C/f1477162972d4ec9b21f5e9f23acf144 2023-03-29 03:55:55,694 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/A/0cc29faa019e4fef853c61dd1783e531 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/0cc29faa019e4fef853c61dd1783e531 2023-03-29 03:55:55,698 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/0cc29faa019e4fef853c61dd1783e531, entries=250, sequenceid=292, filesize=16.7 K 2023-03-29 03:55:55,699 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/B/6460c1d108414bbfab636286877c294e as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/6460c1d108414bbfab636286877c294e 2023-03-29 03:55:55,700 DEBUG [Thread-3261] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:55,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:55,703 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:55,704 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/6460c1d108414bbfab636286877c294e, entries=150, sequenceid=292, filesize=11.9 K 2023-03-29 03:55:55,705 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/C/f1477162972d4ec9b21f5e9f23acf144 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/f1477162972d4ec9b21f5e9f23acf144 2023-03-29 03:55:55,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:55,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:55,705 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:55,705 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:55,705 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:55,705 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:55,706 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:55,706 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:55,707 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:55,707 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:55,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:55,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:55,707 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:55,708 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:55,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:55,708 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:55,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:55,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:55,708 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:55,713 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:55,713 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:55,713 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:55,713 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:55,713 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/f1477162972d4ec9b21f5e9f23acf144, entries=150, sequenceid=292, filesize=11.9 K 2023-03-29 03:55:55,713 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:55,714 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:55,714 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:55,714 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:55,714 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. started... 2023-03-29 03:55:55,714 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. as already flushing 2023-03-29 03:55:55,714 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:55,714 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:55,715 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:55,715 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:55,715 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:55,715 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:55,715 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=100.63 KB/103050 for 0337be0f939be2a2b593eefecee32758 in 80ms, sequenceid=292, compaction requested=true 2023-03-29 03:55:55,715 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 0337be0f939be2a2b593eefecee32758: 2023-03-29 03:55:55,715 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:55:55,715 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-29 03:55:55,715 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-29 03:55:55,715 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:55:55,715 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:55:55,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on 0337be0f939be2a2b593eefecee32758 2023-03-29 03:55:55,715 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 0337be0f939be2a2b593eefecee32758 3/3 column families, dataSize=107.34 KB heapSize=295.92 KB 2023-03-29 03:55:55,715 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0337be0f939be2a2b593eefecee32758, store=A 2023-03-29 03:55:55,715 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-29 03:55:55,716 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0337be0f939be2a2b593eefecee32758, store=B 2023-03-29 03:55:55,716 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-29 03:55:55,716 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0337be0f939be2a2b593eefecee32758, store=C 2023-03-29 03:55:55,716 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-29 03:55:55,716 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37056 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-29 03:55:55,716 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:55,716 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1912): 0337be0f939be2a2b593eefecee32758/B is initiating minor compaction (all files) 2023-03-29 03:55:55,716 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0337be0f939be2a2b593eefecee32758/B in TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:55,716 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/bfabbcc60fdf4b76bf1701a2dd4b285d, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/02125a7e4e2844d59f622f66e4a647ed, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/6460c1d108414bbfab636286877c294e] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp, totalSize=36.2 K 2023-03-29 03:55:55,717 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 41936 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-29 03:55:55,717 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1912): 0337be0f939be2a2b593eefecee32758/A is initiating minor compaction (all files) 2023-03-29 03:55:55,717 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 0337be0f939be2a2b593eefecee32758/A in TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:55,717 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting bfabbcc60fdf4b76bf1701a2dd4b285d, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=243, earliestPutTs=1680062153713 2023-03-29 03:55:55,717 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/68f036894ad443fdb6aa41eb6bd4b127, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/9cb1db4d28db4c4884d5c646b4b2be8c, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/0cc29faa019e4fef853c61dd1783e531] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp, totalSize=41.0 K 2023-03-29 03:55:55,717 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 02125a7e4e2844d59f622f66e4a647ed, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=281, earliestPutTs=1680062154465 2023-03-29 03:55:55,718 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 68f036894ad443fdb6aa41eb6bd4b127, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=243, earliestPutTs=1680062153713 2023-03-29 03:55:55,718 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 9cb1db4d28db4c4884d5c646b4b2be8c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=281, earliestPutTs=1680062154465 2023-03-29 03:55:55,718 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 6460c1d108414bbfab636286877c294e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=292, earliestPutTs=1680062155621 2023-03-29 03:55:55,718 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 0cc29faa019e4fef853c61dd1783e531, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=292, earliestPutTs=1680062155621 2023-03-29 03:55:55,720 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:55,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:55,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:55,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:55,720 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:55,720 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:55,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:55,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:55,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:55,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:55,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:55,725 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:55,725 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1510f73a[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:55,725 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:55,725 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:55,726 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] throttle.PressureAwareThroughputController(145): 0337be0f939be2a2b593eefecee32758#B#compaction#495 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:55:55,726 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0337be0f939be2a2b593eefecee32758#A#compaction#494 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:55:55,728 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:55,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:55,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:55,728 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:55,728 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:55,728 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:55,729 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:55,729 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:55,731 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:55,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:55,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:55,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:55,731 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:55,731 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:55,731 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:55,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:55,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:55,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:55,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:55,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:55,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:55,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:55,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:55,734 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:55,734 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:55,734 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:55,734 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:55,734 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:55,734 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:55,735 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:55,735 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:55,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:55,736 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:55,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:55,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:55,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:55,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:55,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:55,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:55,736 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:55,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:55,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:55,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:55,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:55,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:55,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:55,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:55,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:55,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:55,748 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:55,751 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:55,751 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:55,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:55,751 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:55,751 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:55,751 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:55,751 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:55,751 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:55,751 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:55,751 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:55,751 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:55,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:55,752 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:55,751 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:55,752 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:55,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:55,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:55,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41942 deadline: 1680062215742, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:55,752 DEBUG [Thread-3261] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:55,752 DEBUG [Thread-3261] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:55,756 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:55,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42004 deadline: 1680062215743, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:55,757 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:55,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42010 deadline: 1680062215751, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:55,760 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=324 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/A/d3b472e3f9754511a3c555392a7286a0 2023-03-29 03:55:55,761 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:55,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42024 deadline: 1680062215752, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:55,766 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/A/46961c48b11f4222a0a52f50c284ed12 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/46961c48b11f4222a0a52f50c284ed12 2023-03-29 03:55:55,770 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0337be0f939be2a2b593eefecee32758/A of 0337be0f939be2a2b593eefecee32758 into 46961c48b11f4222a0a52f50c284ed12(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:55:55,770 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0337be0f939be2a2b593eefecee32758: 2023-03-29 03:55:55,770 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758., storeName=0337be0f939be2a2b593eefecee32758/A, priority=13, startTime=1680062155715; duration=0sec 2023-03-29 03:55:55,770 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:55:55,771 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-29 03:55:55,773 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37056 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-29 03:55:55,773 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1912): 0337be0f939be2a2b593eefecee32758/C is initiating minor compaction (all files) 2023-03-29 03:55:55,773 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 0337be0f939be2a2b593eefecee32758/C in TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:55,773 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/eb4a7aafeb7b4e068c8b7db465c15f54, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/5b69893bd5314bc88f715dbae7065fae, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/f1477162972d4ec9b21f5e9f23acf144] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp, totalSize=36.2 K 2023-03-29 03:55:55,774 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting eb4a7aafeb7b4e068c8b7db465c15f54, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=243, earliestPutTs=1680062153713 2023-03-29 03:55:55,774 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 5b69893bd5314bc88f715dbae7065fae, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=281, earliestPutTs=1680062154465 2023-03-29 03:55:55,774 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting f1477162972d4ec9b21f5e9f23acf144, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=292, earliestPutTs=1680062155621 2023-03-29 03:55:55,778 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/B/0167580ea68e417dbca64eec84b86414 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/0167580ea68e417dbca64eec84b86414 2023-03-29 03:55:55,783 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0337be0f939be2a2b593eefecee32758/B of 0337be0f939be2a2b593eefecee32758 into 0167580ea68e417dbca64eec84b86414(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:55:55,783 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0337be0f939be2a2b593eefecee32758: 2023-03-29 03:55:55,783 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758., storeName=0337be0f939be2a2b593eefecee32758/B, priority=13, startTime=1680062155715; duration=0sec 2023-03-29 03:55:55,783 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:55:55,787 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0337be0f939be2a2b593eefecee32758#C#compaction#497 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:55:55,852 DEBUG [Thread-3261] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:55,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:55,856 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:55,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:55,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:55,860 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:55,860 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:55,860 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:55,860 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:55,860 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:55,861 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:55,861 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:55,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41942 deadline: 1680062215855, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:55,862 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:55,862 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:55,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:55,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:55,863 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:55,863 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:55,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:55,863 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:55,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:55,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:55,864 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:55,864 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:55,864 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:55,864 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:55,864 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:55,864 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:55,867 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:55,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42004 deadline: 1680062215858, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:55,867 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:55,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42010 deadline: 1680062215858, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:55,868 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:55,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42024 deadline: 1680062215863, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:55,872 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:55,872 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:55,872 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:55,872 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. started... 2023-03-29 03:55:55,872 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. as already flushing 2023-03-29 03:55:55,872 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:55,872 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:55,872 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:55,872 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:55,872 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:55,872 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:55,875 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:55,875 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:55,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:55,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:55,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:55,875 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:55,875 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:55,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:55,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:55,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:55,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:55,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:55,877 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:55,877 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@763cedec[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:55,877 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:55,877 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:55,878 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:55,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:55,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:55,878 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:55,878 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:55,878 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:55,878 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:55,878 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:55,880 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:55,880 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:55,880 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:55,880 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:55,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:55,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:55,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:55,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:55,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:55,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:55,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:55,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:55,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:55,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:55,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:55,883 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:55,883 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:55,883 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:55,883 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:55,883 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:55,883 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:55,885 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:55,885 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:55,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:55,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:55,885 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:55,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:55,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:55,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:55,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:55,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:55,885 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:55,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:55,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:55,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:55,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:55,886 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:55,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:55,889 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:55,889 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:55,889 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:55,889 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:55,889 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:55,889 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:55,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:55,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:55,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:55,889 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:55,889 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:55,889 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:55,889 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:55,889 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:55,889 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:55,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:55,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:55,890 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:55:55,890 DEBUG [Thread-3261] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:55,890 DEBUG [Thread-3261] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:55,990 DEBUG [Thread-3261] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:55,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:55,992 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:55,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:55,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:55,993 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:55,993 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:55,994 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:55,994 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:55,994 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:55,994 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:55,995 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:55,995 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:55,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:55,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:55,996 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:55,996 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:55,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:55,996 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:55,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:55,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:55,996 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:55,997 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:55,997 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:55,997 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:55,997 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:55,997 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:55,997 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:55,997 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:55,997 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:55,997 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. started... 2023-03-29 03:55:55,997 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. as already flushing 2023-03-29 03:55:55,997 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:55,997 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:55,997 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:55,997 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:55,997 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:55,997 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:55,999 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:55,999 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:55,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:55,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:55,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:55,999 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:55,999 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:55,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:56,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:56,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:56,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:56,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:56,001 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:56,001 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@578ce7d4[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:56,001 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:56,001 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:56,002 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:56,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:56,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:56,002 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:56,002 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:56,002 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:56,002 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:56,002 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:56,004 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:56,004 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:56,004 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:56,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:56,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:56,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:56,005 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:56,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:56,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:56,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:56,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:56,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:56,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:56,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:56,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:56,007 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:56,007 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:56,007 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:56,007 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:56,007 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:56,007 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:56,008 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:56,008 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:56,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:56,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:56,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:56,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:56,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:56,009 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:56,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:56,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:56,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:56,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:56,011 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:56,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:56,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:56,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:56,012 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:56,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:56,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:56,024 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:56,024 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:56,024 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:56,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:56,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:56,024 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:56,024 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:56,024 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:56,024 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:56,024 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:56,024 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:56,025 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:56,024 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:56,025 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:56,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:56,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:56,025 DEBUG [Thread-3261] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:56,025 DEBUG [Thread-3261] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:56,025 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:55:56,066 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:56,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41942 deadline: 1680062216063, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:56,077 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:56,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42010 deadline: 1680062216069, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:56,078 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:56,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42024 deadline: 1680062216069, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:56,078 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:56,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42004 deadline: 1680062216069, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:56,126 DEBUG [Thread-3261] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:56,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:56,128 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:56,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:56,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:56,129 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:56,129 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:56,129 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:56,129 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:56,130 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:56,130 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:56,132 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:56,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:56,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:56,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:56,133 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:56,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:56,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:56,133 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:56,133 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:56,133 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:56,134 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:56,134 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:56,134 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:56,134 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:56,134 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:56,134 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:56,134 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:56,134 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:56,134 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:56,134 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. started... 2023-03-29 03:55:56,134 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. as already flushing 2023-03-29 03:55:56,134 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:56,134 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:56,134 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:56,134 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:56,134 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:56,134 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:56,136 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:56,136 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:56,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:56,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:56,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:56,136 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:56,136 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:56,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:56,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:56,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:56,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:56,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:56,138 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:56,138 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a6fd3a9[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:56,138 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:56,138 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:56,139 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:56,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:56,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:56,139 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:56,139 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:56,139 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:56,140 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:56,140 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:56,141 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:56,141 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:56,141 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:56,141 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:56,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:56,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:56,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:56,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:56,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:56,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:56,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:56,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:56,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:56,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:56,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:56,143 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:56,143 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:56,143 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:56,143 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:56,143 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:56,143 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:56,146 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:56,146 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:56,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:56,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:56,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:56,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:56,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:56,146 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:56,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:56,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:56,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:56,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:56,148 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:56,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:56,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:56,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:56,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:56,150 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:56,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:56,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:56,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:56,152 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:56,152 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:56,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:56,152 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:56,152 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:56,152 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:56,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:56,152 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:56,153 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:56,153 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:56,153 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:56,153 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:56,152 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:56,153 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:56,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:56,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:56,153 DEBUG [Thread-3261] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:56,153 DEBUG [Thread-3261] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:56,185 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=324 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/B/4a490c12a32b450b89b4fc545245cad8 2023-03-29 03:55:56,200 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/C/c47493d2a47542af926eb7208eb11c00 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/c47493d2a47542af926eb7208eb11c00 2023-03-29 03:55:56,210 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0337be0f939be2a2b593eefecee32758/C of 0337be0f939be2a2b593eefecee32758 into c47493d2a47542af926eb7208eb11c00(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:55:56,211 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0337be0f939be2a2b593eefecee32758: 2023-03-29 03:55:56,211 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758., storeName=0337be0f939be2a2b593eefecee32758/C, priority=13, startTime=1680062155715; duration=0sec 2023-03-29 03:55:56,211 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:55:56,211 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=324 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/C/b86beaf8145a40dc9dac63307bcdf306 2023-03-29 03:55:56,223 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/A/d3b472e3f9754511a3c555392a7286a0 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/d3b472e3f9754511a3c555392a7286a0 2023-03-29 03:55:56,228 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/d3b472e3f9754511a3c555392a7286a0, entries=350, sequenceid=324, filesize=21.4 K 2023-03-29 03:55:56,229 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/B/4a490c12a32b450b89b4fc545245cad8 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/4a490c12a32b450b89b4fc545245cad8 2023-03-29 03:55:56,233 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/4a490c12a32b450b89b4fc545245cad8, entries=150, sequenceid=324, filesize=11.9 K 2023-03-29 03:55:56,234 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/C/b86beaf8145a40dc9dac63307bcdf306 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/b86beaf8145a40dc9dac63307bcdf306 2023-03-29 03:55:56,237 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/b86beaf8145a40dc9dac63307bcdf306, entries=150, sequenceid=324, filesize=11.9 K 2023-03-29 03:55:56,238 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~295.88 KB/302976, currentSize=73.80 KB/75570 for 0337be0f939be2a2b593eefecee32758 in 523ms, sequenceid=324, compaction requested=false 2023-03-29 03:55:56,238 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 0337be0f939be2a2b593eefecee32758: 2023-03-29 03:55:56,253 DEBUG [Thread-3261] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:56,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:56,259 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:56,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:56,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:56,268 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:56,268 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:56,268 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:56,268 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:56,269 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:56,269 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:56,270 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:56,270 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:56,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:56,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:56,270 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:56,270 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:56,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:56,271 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:56,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:56,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:56,273 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:56,280 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:56,280 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:56,280 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:56,280 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:56,280 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:56,286 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:56,287 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:56,287 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:56,287 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. started... 2023-03-29 03:55:56,287 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0337be0f939be2a2b593eefecee32758 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-03-29 03:55:56,287 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0337be0f939be2a2b593eefecee32758, store=A 2023-03-29 03:55:56,287 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:55:56,287 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0337be0f939be2a2b593eefecee32758, store=B 2023-03-29 03:55:56,289 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:55:56,289 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0337be0f939be2a2b593eefecee32758, store=C 2023-03-29 03:55:56,289 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:55:56,302 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=341 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/A/8ab5930905054ddbb64101af67bdaa9c 2023-03-29 03:55:56,312 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=341 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/B/a1dd3e80578145538dd856018625ba93 2023-03-29 03:55:56,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on 0337be0f939be2a2b593eefecee32758 2023-03-29 03:55:56,374 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. as already flushing 2023-03-29 03:55:56,412 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:56,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42010 deadline: 1680062216403, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:56,413 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:56,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42004 deadline: 1680062216405, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:56,419 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:56,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42024 deadline: 1680062216409, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:56,419 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:56,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41942 deadline: 1680062216410, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:56,520 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:56,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42004 deadline: 1680062216517, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:56,520 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:56,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41942 deadline: 1680062216520, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:56,521 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:56,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42010 deadline: 1680062216520, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:56,521 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:56,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42024 deadline: 1680062216521, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:56,730 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:56,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42004 deadline: 1680062216723, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:56,730 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:56,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41942 deadline: 1680062216723, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:56,731 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:56,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42024 deadline: 1680062216723, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:56,731 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:56,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42010 deadline: 1680062216723, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:56,733 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=341 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/C/ea547bd02fc34e77aa006ef48572d607 2023-03-29 03:55:56,740 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/A/8ab5930905054ddbb64101af67bdaa9c as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/8ab5930905054ddbb64101af67bdaa9c 2023-03-29 03:55:56,745 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/8ab5930905054ddbb64101af67bdaa9c, entries=150, sequenceid=341, filesize=11.9 K 2023-03-29 03:55:56,746 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/B/a1dd3e80578145538dd856018625ba93 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/a1dd3e80578145538dd856018625ba93 2023-03-29 03:55:56,765 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/a1dd3e80578145538dd856018625ba93, entries=150, sequenceid=341, filesize=11.9 K 2023-03-29 03:55:56,766 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/C/ea547bd02fc34e77aa006ef48572d607 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/ea547bd02fc34e77aa006ef48572d607 2023-03-29 03:55:56,779 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/ea547bd02fc34e77aa006ef48572d607, entries=150, sequenceid=341, filesize=11.9 K 2023-03-29 03:55:56,780 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 0337be0f939be2a2b593eefecee32758 in 493ms, sequenceid=341, compaction requested=true 2023-03-29 03:55:56,780 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0337be0f939be2a2b593eefecee32758: 2023-03-29 03:55:56,780 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:56,780 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:56,780 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:56,780 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:56,780 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:56,780 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:56,782 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:56,782 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:56,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:56,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:56,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:56,782 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:56,782 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:56,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:56,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:56,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:56,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:56,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:56,785 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:56,785 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@785440f9[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:56,785 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:56,785 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:56,792 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:56,792 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:56,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:56,792 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:56,792 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:56,793 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:56,794 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:56,794 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:56,795 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:56,795 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:56,796 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:56,796 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:56,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:56,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:56,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:56,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:56,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:56,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:56,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:56,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:56,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:56,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:56,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:56,798 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:56,798 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:56,798 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:56,798 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:56,798 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:56,798 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:56,801 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:56,801 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:56,801 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:56,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:56,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:56,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:56,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:56,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:56,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:56,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:56,802 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:56,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:56,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:56,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:56,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:56,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:56,804 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:56,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:56,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:56,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:56,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:56,807 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:56,807 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:56,807 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:56,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:56,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:56,807 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:56,807 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:56,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:56,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:56,807 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:56,808 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:56,808 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:56,808 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:56,808 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:56,808 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:56,807 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:56,808 DEBUG [Thread-3261] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:56,808 DEBUG [Thread-3261] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:56,808 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:55:56,908 DEBUG [Thread-3261] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:56,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:56,910 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:56,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:56,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:56,912 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:56,912 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:56,912 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:56,912 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:56,912 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:56,912 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:56,915 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:56,915 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:56,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:56,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:56,915 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:56,915 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:56,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:56,916 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:56,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:56,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:56,916 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:56,916 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:56,916 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:56,916 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:56,916 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:56,917 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:56,920 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:56,920 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:56,920 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:56,920 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. started... 2023-03-29 03:55:56,920 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0337be0f939be2a2b593eefecee32758 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-03-29 03:55:56,920 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0337be0f939be2a2b593eefecee32758, store=A 2023-03-29 03:55:56,920 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:55:56,921 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0337be0f939be2a2b593eefecee32758, store=B 2023-03-29 03:55:56,921 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:55:56,921 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0337be0f939be2a2b593eefecee32758, store=C 2023-03-29 03:55:56,921 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:55:56,955 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=359 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/A/3927b7ab768b4f0fbfc2cef0f955a8cd 2023-03-29 03:55:56,982 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=359 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/B/a864856115004230843d2ec088e64dea 2023-03-29 03:55:57,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on 0337be0f939be2a2b593eefecee32758 2023-03-29 03:55:57,036 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. as already flushing 2023-03-29 03:55:57,037 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=359 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/C/0ee4e9ae9a1140c4a619e0d04219eee3 2023-03-29 03:55:57,042 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/A/3927b7ab768b4f0fbfc2cef0f955a8cd as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/3927b7ab768b4f0fbfc2cef0f955a8cd 2023-03-29 03:55:57,045 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/3927b7ab768b4f0fbfc2cef0f955a8cd, entries=150, sequenceid=359, filesize=11.9 K 2023-03-29 03:55:57,045 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/B/a864856115004230843d2ec088e64dea as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/a864856115004230843d2ec088e64dea 2023-03-29 03:55:57,053 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/a864856115004230843d2ec088e64dea, entries=150, sequenceid=359, filesize=11.9 K 2023-03-29 03:55:57,054 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/C/0ee4e9ae9a1140c4a619e0d04219eee3 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/0ee4e9ae9a1140c4a619e0d04219eee3 2023-03-29 03:55:57,064 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/0ee4e9ae9a1140c4a619e0d04219eee3, entries=150, sequenceid=359, filesize=11.9 K 2023-03-29 03:55:57,064 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 0337be0f939be2a2b593eefecee32758 in 144ms, sequenceid=359, compaction requested=true 2023-03-29 03:55:57,064 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0337be0f939be2a2b593eefecee32758: 2023-03-29 03:55:57,064 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:57,065 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:57,065 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:57,065 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:57,068 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:57,068 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:57,069 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:57,069 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:57,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:57,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:57,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:57,070 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:57,070 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:57,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:57,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:57,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:57,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:57,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:57,071 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:57,071 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69169192[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:57,071 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:57,071 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:57,072 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:57,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:57,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:57,073 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:57,073 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:57,073 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:57,073 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:57,073 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:57,074 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:57,074 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:57,074 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:57,074 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:57,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:57,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:57,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:57,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:57,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:57,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:57,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:57,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:57,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:57,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:57,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:57,077 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:57,077 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:57,077 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:57,077 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:57,077 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:57,077 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:57,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on 0337be0f939be2a2b593eefecee32758 2023-03-29 03:55:57,079 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 0337be0f939be2a2b593eefecee32758 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-03-29 03:55:57,079 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0337be0f939be2a2b593eefecee32758, store=A 2023-03-29 03:55:57,079 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:55:57,079 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0337be0f939be2a2b593eefecee32758, store=B 2023-03-29 03:55:57,079 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:55:57,079 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0337be0f939be2a2b593eefecee32758, store=C 2023-03-29 03:55:57,079 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:55:57,083 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:57,083 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:57,083 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:57,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:57,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:57,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:57,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:57,084 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:57,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:57,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:57,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:57,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:57,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:57,085 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:57,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:57,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:57,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:57,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:57,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:57,089 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:57,089 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:57,089 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:57,089 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:57,089 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:57,089 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:57,089 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:57,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:57,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:57,089 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:57,090 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:55:57,089 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:57,091 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:57,091 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:57,090 DEBUG [Thread-3261] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:57,091 DEBUG [Thread-3261] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:57,089 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:57,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:57,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:57,106 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=376 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/A/28762f05a76f4389ae4e2b94d00c18d9 2023-03-29 03:55:57,121 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:57,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42024 deadline: 1680062217109, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:57,122 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:57,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42004 deadline: 1680062217115, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:57,127 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:57,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42010 deadline: 1680062217116, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:57,132 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:57,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41942 deadline: 1680062217122, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:57,134 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=376 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/B/623e374564df4490ab343b2bedecc9ef 2023-03-29 03:55:57,162 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=376 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/C/89e15f533c40459883faf24b666e822c 2023-03-29 03:55:57,167 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/A/28762f05a76f4389ae4e2b94d00c18d9 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/28762f05a76f4389ae4e2b94d00c18d9 2023-03-29 03:55:57,171 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/28762f05a76f4389ae4e2b94d00c18d9, entries=150, sequenceid=376, filesize=11.9 K 2023-03-29 03:55:57,172 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/B/623e374564df4490ab343b2bedecc9ef as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/623e374564df4490ab343b2bedecc9ef 2023-03-29 03:55:57,177 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/623e374564df4490ab343b2bedecc9ef, entries=150, sequenceid=376, filesize=11.9 K 2023-03-29 03:55:57,178 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/C/89e15f533c40459883faf24b666e822c as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/89e15f533c40459883faf24b666e822c 2023-03-29 03:55:57,188 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/89e15f533c40459883faf24b666e822c, entries=150, sequenceid=376, filesize=11.9 K 2023-03-29 03:55:57,189 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 0337be0f939be2a2b593eefecee32758 in 110ms, sequenceid=376, compaction requested=true 2023-03-29 03:55:57,189 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 0337be0f939be2a2b593eefecee32758: 2023-03-29 03:55:57,189 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:55:57,189 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-29 03:55:57,189 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:55:57,189 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-29 03:55:57,189 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:55:57,191 DEBUG [Thread-3261] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:57,192 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61680 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-29 03:55:57,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:57,192 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1912): 0337be0f939be2a2b593eefecee32758/B is initiating minor compaction (all files) 2023-03-29 03:55:57,192 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0337be0f939be2a2b593eefecee32758/B in TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:57,192 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/0167580ea68e417dbca64eec84b86414, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/4a490c12a32b450b89b4fc545245cad8, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/a1dd3e80578145538dd856018625ba93, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/a864856115004230843d2ec088e64dea, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/623e374564df4490ab343b2bedecc9ef] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp, totalSize=60.2 K 2023-03-29 03:55:57,194 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:57,194 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 71444 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-29 03:55:57,194 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1912): 0337be0f939be2a2b593eefecee32758/A is initiating minor compaction (all files) 2023-03-29 03:55:57,194 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 0337be0f939be2a2b593eefecee32758/A in TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:57,194 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/46961c48b11f4222a0a52f50c284ed12, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/d3b472e3f9754511a3c555392a7286a0, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/8ab5930905054ddbb64101af67bdaa9c, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/3927b7ab768b4f0fbfc2cef0f955a8cd, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/28762f05a76f4389ae4e2b94d00c18d9] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp, totalSize=69.8 K 2023-03-29 03:55:57,194 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 0167580ea68e417dbca64eec84b86414, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=292, earliestPutTs=1680062155621 2023-03-29 03:55:57,195 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 4a490c12a32b450b89b4fc545245cad8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=324, earliestPutTs=1680062155711 2023-03-29 03:55:57,196 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 46961c48b11f4222a0a52f50c284ed12, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=292, earliestPutTs=1680062155621 2023-03-29 03:55:57,196 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting a1dd3e80578145538dd856018625ba93, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=341, earliestPutTs=1680062155737 2023-03-29 03:55:57,196 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting d3b472e3f9754511a3c555392a7286a0, keycount=350, bloomtype=ROW, size=21.4 K, encoding=NONE, compression=NONE, seqNum=324, earliestPutTs=1680062155659 2023-03-29 03:55:57,196 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 8ab5930905054ddbb64101af67bdaa9c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=341, earliestPutTs=1680062155737 2023-03-29 03:55:57,197 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 3927b7ab768b4f0fbfc2cef0f955a8cd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=359, earliestPutTs=1680062156403 2023-03-29 03:55:57,197 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 28762f05a76f4389ae4e2b94d00c18d9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=376, earliestPutTs=1680062157058 2023-03-29 03:55:57,197 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting a864856115004230843d2ec088e64dea, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=359, earliestPutTs=1680062156403 2023-03-29 03:55:57,198 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 623e374564df4490ab343b2bedecc9ef, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=376, earliestPutTs=1680062157058 2023-03-29 03:55:57,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:57,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:57,208 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:57,208 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:57,208 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:57,208 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:57,208 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:57,209 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:57,210 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:57,210 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:57,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:57,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:57,211 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:57,211 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:57,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:57,211 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:57,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:57,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:57,212 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:57,219 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:57,219 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:57,220 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:57,220 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:57,220 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:57,230 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0337be0f939be2a2b593eefecee32758#A#compaction#508 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:55:57,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on 0337be0f939be2a2b593eefecee32758 2023-03-29 03:55:57,231 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 0337be0f939be2a2b593eefecee32758 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-03-29 03:55:57,232 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0337be0f939be2a2b593eefecee32758, store=A 2023-03-29 03:55:57,232 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:55:57,232 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0337be0f939be2a2b593eefecee32758, store=B 2023-03-29 03:55:57,232 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:55:57,232 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0337be0f939be2a2b593eefecee32758, store=C 2023-03-29 03:55:57,232 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:55:57,238 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] throttle.PressureAwareThroughputController(145): 0337be0f939be2a2b593eefecee32758#B#compaction#509 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:55:57,240 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:57,240 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:57,241 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:57,241 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. started... 2023-03-29 03:55:57,241 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. as already flushing 2023-03-29 03:55:57,241 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:57,241 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:57,241 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:57,241 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:57,245 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:57,245 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:57,254 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:57,254 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:57,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:57,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:57,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:57,254 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:57,255 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:57,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:57,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:57,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:57,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:57,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:57,256 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:57,256 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@59946923[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:57,256 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:57,256 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:57,257 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:57,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:57,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:57,258 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:57,258 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:57,258 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:57,258 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:57,258 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:57,259 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:57,259 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:57,259 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:57,259 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:57,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:57,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:57,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:57,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:57,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:57,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:57,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:57,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:57,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:57,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:57,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:57,262 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:57,262 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:57,262 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:57,262 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:57,262 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:57,262 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:57,264 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:57,264 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:57,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:57,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:57,264 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:57,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:57,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:57,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:57,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:57,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:57,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:57,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:57,265 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:57,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:57,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:57,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:57,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:57,266 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:57,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:57,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:57,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:57,271 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:57,272 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:57,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:57,272 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:57,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:57,272 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:57,272 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:57,272 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:57,272 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:57,272 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:57,272 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:57,272 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:57,272 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:57,272 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:57,272 DEBUG [Thread-3261] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:57,272 DEBUG [Thread-3261] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:57,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:57,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:57,279 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:57,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42004 deadline: 1680062217269, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:57,284 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:57,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42010 deadline: 1680062217270, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:57,287 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:57,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42024 deadline: 1680062217278, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:57,293 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:57,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41942 deadline: 1680062217280, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:57,330 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=392 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/A/4db6ee64cb5a4af4a1e13f51ebe3176f 2023-03-29 03:55:57,373 DEBUG [Thread-3261] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:57,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:57,375 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:57,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:57,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:57,388 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:57,388 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:57,388 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:57,388 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:57,388 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:57,389 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:57,390 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:57,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:57,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:57,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:57,391 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:57,391 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:57,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:57,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:57,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42004 deadline: 1680062217381, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:57,391 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:57,392 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:57,392 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:57,392 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:57,394 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:57,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42010 deadline: 1680062217386, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:57,394 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:57,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42024 deadline: 1680062217389, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:57,400 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:57,400 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:57,400 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:57,400 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:57,400 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:57,402 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:57,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41942 deadline: 1680062217394, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:57,404 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:57,404 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:57,404 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:57,404 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. started... 2023-03-29 03:55:57,404 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. as already flushing 2023-03-29 03:55:57,404 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:57,405 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:57,405 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:57,405 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:57,405 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:57,405 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:57,406 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:57,406 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:57,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:57,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:57,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:57,407 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:57,407 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:57,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:57,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:57,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:57,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:57,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:57,409 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:57,409 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4df20b23[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:57,409 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:57,410 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:57,411 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:57,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:57,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:57,411 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:57,411 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:57,411 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:57,411 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:57,411 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:57,413 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:57,413 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:57,413 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:57,413 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:57,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:57,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:57,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:57,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:57,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:57,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:57,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:57,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:57,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:57,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:57,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:57,415 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:57,415 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:57,415 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:57,415 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:57,415 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:57,415 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:57,417 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:57,417 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:57,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:57,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:57,417 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:57,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:57,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:57,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:57,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:57,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:57,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:57,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:57,418 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:57,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:57,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:57,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:57,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:57,420 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:57,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:57,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:57,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:57,422 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:57,422 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:57,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:57,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:57,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:57,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:57,423 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:57,423 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:57,423 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:57,423 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:57,423 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:57,423 DEBUG [Thread-3261] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:57,423 DEBUG [Thread-3261] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:57,423 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:57,423 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:57,424 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:57,424 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:57,424 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:57,523 DEBUG [Thread-3261] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:57,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:57,526 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:57,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:57,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:57,531 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:57,532 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:57,532 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:57,532 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:57,532 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:57,532 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:57,534 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:57,534 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:57,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:57,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:57,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:57,534 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:57,534 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:57,535 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:57,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:57,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:57,535 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:57,536 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:57,536 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:57,536 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:57,536 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:57,536 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:57,544 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:57,544 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:57,544 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:57,544 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. started... 2023-03-29 03:55:57,545 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. as already flushing 2023-03-29 03:55:57,545 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:57,545 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:57,545 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:57,545 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:57,545 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:57,545 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:57,548 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:57,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:57,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:57,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:57,548 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:57,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:57,549 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:57,549 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:57,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:57,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:57,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:57,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:57,552 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:57,552 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@177bc9ec[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:57,552 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:57,552 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:57,553 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:57,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:57,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:57,553 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:57,553 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:57,553 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:57,554 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:57,554 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:57,560 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:57,561 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:57,561 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:57,562 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:57,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:57,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:57,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:57,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:57,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:57,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:57,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:57,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:57,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:57,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:57,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:57,570 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:57,570 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:57,570 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:57,570 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:57,570 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:57,570 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:57,572 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:57,572 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:57,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:57,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:57,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:57,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:57,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:57,573 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:57,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:57,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:57,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:57,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:57,574 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:57,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:57,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:57,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:57,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:57,578 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:57,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:57,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:57,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:57,582 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:57,582 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:57,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:57,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:57,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:57,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:57,583 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:57,583 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:57,583 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:57,583 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:57,583 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:57,583 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:57,583 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:57,583 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:57,583 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:57,583 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:57,584 DEBUG [Thread-3261] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:57,584 DEBUG [Thread-3261] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:57,603 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:57,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42004 deadline: 1680062217593, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:57,606 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:57,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42010 deadline: 1680062217595, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:57,606 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:57,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42024 deadline: 1680062217597, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:57,617 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:57,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41942 deadline: 1680062217606, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:57,684 DEBUG [Thread-3261] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:57,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:57,690 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:57,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:57,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:57,696 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:57,696 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:57,696 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:57,696 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:57,697 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:57,697 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:57,698 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:57,698 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:57,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:57,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:57,699 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:57,699 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:57,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:57,700 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:57,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:57,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:57,701 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:57,702 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/B/844eafdd81e846a7a821b9853a7a670c as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/844eafdd81e846a7a821b9853a7a670c 2023-03-29 03:55:57,706 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 0337be0f939be2a2b593eefecee32758/B of 0337be0f939be2a2b593eefecee32758 into 844eafdd81e846a7a821b9853a7a670c(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:55:57,706 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0337be0f939be2a2b593eefecee32758: 2023-03-29 03:55:57,706 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758., storeName=0337be0f939be2a2b593eefecee32758/B, priority=11, startTime=1680062157189; duration=0sec 2023-03-29 03:55:57,706 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:55:57,707 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-29 03:55:57,708 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61680 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-29 03:55:57,708 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1912): 0337be0f939be2a2b593eefecee32758/C is initiating minor compaction (all files) 2023-03-29 03:55:57,708 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0337be0f939be2a2b593eefecee32758/C in TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:57,708 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/c47493d2a47542af926eb7208eb11c00, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/b86beaf8145a40dc9dac63307bcdf306, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/ea547bd02fc34e77aa006ef48572d607, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/0ee4e9ae9a1140c4a619e0d04219eee3, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/89e15f533c40459883faf24b666e822c] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp, totalSize=60.2 K 2023-03-29 03:55:57,709 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting c47493d2a47542af926eb7208eb11c00, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=292, earliestPutTs=1680062155621 2023-03-29 03:55:57,709 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting b86beaf8145a40dc9dac63307bcdf306, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=324, earliestPutTs=1680062155711 2023-03-29 03:55:57,709 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting ea547bd02fc34e77aa006ef48572d607, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=341, earliestPutTs=1680062155737 2023-03-29 03:55:57,710 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 0ee4e9ae9a1140c4a619e0d04219eee3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=359, earliestPutTs=1680062156403 2023-03-29 03:55:57,710 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 89e15f533c40459883faf24b666e822c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=376, earliestPutTs=1680062157058 2023-03-29 03:55:57,713 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:57,713 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:57,713 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:57,713 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:57,714 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:57,714 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:57,714 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:57,714 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:57,714 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. started... 2023-03-29 03:55:57,714 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. as already flushing 2023-03-29 03:55:57,714 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:57,714 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:57,714 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:57,714 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:57,714 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:57,714 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:57,717 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:57,717 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:57,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:57,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:57,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:57,717 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:57,717 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:57,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:57,719 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/A/dfe69b2dd25d4dd0b91387ae0e2d41e4 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/dfe69b2dd25d4dd0b91387ae0e2d41e4 2023-03-29 03:55:57,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:57,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:57,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:57,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:57,721 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:57,721 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a3c5090[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:57,721 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:57,721 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:57,746 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:57,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:57,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:57,746 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:57,746 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:57,746 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:57,747 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] throttle.PressureAwareThroughputController(145): 0337be0f939be2a2b593eefecee32758#C#compaction#512 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:55:57,747 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:57,747 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:57,748 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 0337be0f939be2a2b593eefecee32758/A of 0337be0f939be2a2b593eefecee32758 into dfe69b2dd25d4dd0b91387ae0e2d41e4(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:55:57,749 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0337be0f939be2a2b593eefecee32758: 2023-03-29 03:55:57,749 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758., storeName=0337be0f939be2a2b593eefecee32758/A, priority=11, startTime=1680062157189; duration=0sec 2023-03-29 03:55:57,749 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:55:57,749 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:57,749 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:57,749 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:57,749 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:57,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:57,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:57,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:57,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:57,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:57,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:57,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:57,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:57,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:57,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:57,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:57,751 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:57,751 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:57,751 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:57,751 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:57,751 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:57,751 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:57,753 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:57,753 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:57,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:57,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:57,753 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:57,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:57,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:57,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:57,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:57,754 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:57,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:57,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:57,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:57,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:57,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:57,755 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:57,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:57,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:57,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:57,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:57,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:57,758 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:57,758 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:57,758 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:57,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:57,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:57,758 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:57,758 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:57,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:57,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:57,758 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:57,759 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:57,759 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:57,759 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:57,759 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:57,759 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:57,759 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:57,759 DEBUG [Thread-3261] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:57,759 DEBUG [Thread-3261] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:57,759 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:55:57,768 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/C/ae7f434d973b4520819f2fed2e18375b as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/ae7f434d973b4520819f2fed2e18375b 2023-03-29 03:55:57,768 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=392 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/B/034bb61da4d740b28952a27021b0526c 2023-03-29 03:55:57,772 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 0337be0f939be2a2b593eefecee32758/C of 0337be0f939be2a2b593eefecee32758 into ae7f434d973b4520819f2fed2e18375b(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:55:57,773 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0337be0f939be2a2b593eefecee32758: 2023-03-29 03:55:57,773 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758., storeName=0337be0f939be2a2b593eefecee32758/C, priority=11, startTime=1680062157189; duration=0sec 2023-03-29 03:55:57,773 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:55:57,859 DEBUG [Thread-3261] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:57,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:57,861 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:57,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:57,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:57,862 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:57,862 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:57,862 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:57,862 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:57,863 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:57,863 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:57,865 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:57,865 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:57,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:57,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:57,866 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:57,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:57,866 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:57,866 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:57,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:57,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:57,867 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:57,867 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:57,867 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:57,867 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:57,867 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:57,867 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:57,867 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:57,867 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:57,867 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:57,867 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. started... 2023-03-29 03:55:57,867 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. as already flushing 2023-03-29 03:55:57,867 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:57,869 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:57,869 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:57,869 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:57,869 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:57,869 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:57,870 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:57,870 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:57,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:57,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:57,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:57,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:57,871 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:57,871 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:57,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:57,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:57,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:57,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:57,874 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:57,874 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@301be585[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:57,874 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:57,874 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:57,875 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:57,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:57,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:57,876 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:57,876 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:57,876 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:57,876 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:57,876 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:57,877 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:57,877 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:57,877 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:57,878 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:57,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:57,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:57,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:57,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:57,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:57,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:57,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:57,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:57,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:57,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:57,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:57,882 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:57,882 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:57,882 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:57,882 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:57,882 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:57,882 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:57,887 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:57,887 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:57,887 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:57,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:57,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:57,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:57,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:57,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:57,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:57,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:57,888 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:57,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:57,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:57,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:57,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:57,890 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:57,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:57,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:57,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:57,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:57,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:57,892 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:57,892 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:57,892 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:57,892 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:57,892 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:57,892 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:57,892 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:57,892 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:57,892 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:57,892 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:57,892 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:57,892 DEBUG [Thread-3261] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:57,892 DEBUG [Thread-3261] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:57,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:57,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:57,892 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:57,893 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:55:57,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:57,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:57,905 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:57,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42004 deadline: 1680062217904, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:57,912 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:57,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42010 deadline: 1680062217907, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:57,913 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:57,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42024 deadline: 1680062217909, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:57,925 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:57,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41942 deadline: 1680062217920, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:57,992 DEBUG [Thread-3261] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:57,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:57,997 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:58,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:58,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:58,011 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:58,011 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:58,011 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:58,012 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:58,012 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:58,012 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:58,013 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:58,013 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:58,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:58,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:58,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:58,014 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:58,014 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:58,015 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:58,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:58,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:58,016 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:58,024 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:58,024 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:58,024 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:58,024 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:58,024 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:58,025 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:58,025 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:58,025 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:58,025 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. started... 2023-03-29 03:55:58,025 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. as already flushing 2023-03-29 03:55:58,025 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:58,025 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:58,025 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:58,025 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:58,025 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:58,025 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:58,027 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:58,027 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:58,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:58,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:58,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:58,028 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:58,028 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:58,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:58,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:58,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:58,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:58,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:58,029 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:58,029 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ded5bb6[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:58,029 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:58,030 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:58,031 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:58,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:58,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:58,032 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:58,032 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:58,032 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:58,032 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:58,032 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:58,033 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:58,033 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:58,033 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:58,034 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:58,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:58,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:58,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:58,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:58,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:58,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:58,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:58,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:58,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:58,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:58,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:58,036 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:58,036 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:58,036 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:58,036 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:58,037 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:58,037 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:58,040 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:58,040 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:58,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:58,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:58,040 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:58,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:58,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:58,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:58,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:58,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:58,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:58,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:58,041 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:58,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:58,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:58,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:58,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:58,044 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:58,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:58,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:58,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:58,046 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:58,046 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:58,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:58,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:58,047 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:58,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:58,047 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:58,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:58,047 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:58,047 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:58,047 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:58,047 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:58,047 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:58,047 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:58,047 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:58,047 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:58,048 DEBUG [Thread-3261] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:58,048 DEBUG [Thread-3261] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:58,148 DEBUG [Thread-3261] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:58,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:58,153 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:58,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:58,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:58,156 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:58,156 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:58,156 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:58,156 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:58,158 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:58,158 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:58,160 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:58,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:58,161 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:58,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:58,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:58,161 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:58,161 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:58,161 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:58,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:58,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:58,162 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:58,163 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:58,163 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:58,163 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:58,163 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:58,163 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:58,164 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:58,164 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:58,164 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:58,164 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. started... 2023-03-29 03:55:58,164 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. as already flushing 2023-03-29 03:55:58,164 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:58,164 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:58,164 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:58,164 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:58,164 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:58,164 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:58,166 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:58,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:58,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:58,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:58,166 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:58,166 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:58,166 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:58,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:58,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:58,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:58,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:58,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:58,168 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:58,169 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2410cb4e[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:58,169 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:58,169 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:58,170 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:58,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:58,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:58,170 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:58,170 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:58,170 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:58,174 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:58,175 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:58,175 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:58,175 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:58,175 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:58,175 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:58,175 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:58,175 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:58,175 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:58,175 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:58,175 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:58,177 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:58,177 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:58,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:58,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:58,177 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:58,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:58,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:58,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:58,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:58,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:58,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:58,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:58,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:58,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:58,180 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:58,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:58,181 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:58,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:58,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:58,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:58,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:58,183 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:58,183 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:58,184 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:58,184 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:58,184 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:58,184 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:58,184 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:58,184 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:58,184 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:58,184 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:58,184 DEBUG [Thread-3261] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:58,184 DEBUG [Thread-3261] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:58,184 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:58,185 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:58,185 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:55:58,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:58,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:58,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:58,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:58,191 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=392 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/C/b2d96e8382364215ba9580656d165323 2023-03-29 03:55:58,196 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/A/4db6ee64cb5a4af4a1e13f51ebe3176f as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/4db6ee64cb5a4af4a1e13f51ebe3176f 2023-03-29 03:55:58,199 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/4db6ee64cb5a4af4a1e13f51ebe3176f, entries=200, sequenceid=392, filesize=14.3 K 2023-03-29 03:55:58,200 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/B/034bb61da4d740b28952a27021b0526c as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/034bb61da4d740b28952a27021b0526c 2023-03-29 03:55:58,203 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/034bb61da4d740b28952a27021b0526c, entries=150, sequenceid=392, filesize=11.9 K 2023-03-29 03:55:58,204 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/C/b2d96e8382364215ba9580656d165323 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/b2d96e8382364215ba9580656d165323 2023-03-29 03:55:58,210 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/b2d96e8382364215ba9580656d165323, entries=150, sequenceid=392, filesize=11.9 K 2023-03-29 03:55:58,211 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 0337be0f939be2a2b593eefecee32758 in 980ms, sequenceid=392, compaction requested=false 2023-03-29 03:55:58,211 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 0337be0f939be2a2b593eefecee32758: 2023-03-29 03:55:58,284 DEBUG [Thread-3261] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:58,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:58,286 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:58,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:58,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:58,290 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:58,290 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:58,290 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:58,290 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:58,290 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:58,290 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:58,292 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:58,292 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:58,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:58,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:58,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:58,292 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:58,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:58,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:58,293 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:58,293 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:58,293 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:58,293 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:58,293 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:58,293 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:58,294 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:58,294 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:58,294 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:58,294 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:58,294 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:58,294 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. started... 2023-03-29 03:55:58,294 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0337be0f939be2a2b593eefecee32758 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-03-29 03:55:58,294 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0337be0f939be2a2b593eefecee32758, store=A 2023-03-29 03:55:58,294 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:55:58,295 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0337be0f939be2a2b593eefecee32758, store=B 2023-03-29 03:55:58,295 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:55:58,295 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0337be0f939be2a2b593eefecee32758, store=C 2023-03-29 03:55:58,295 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:55:58,314 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=411 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/A/7141a92d45374ef8920636399cf79454 2023-03-29 03:55:58,336 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=411 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/B/40f05e34f4cf4856930507d82699303b 2023-03-29 03:55:58,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on 0337be0f939be2a2b593eefecee32758 2023-03-29 03:55:58,412 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. as already flushing 2023-03-29 03:55:58,439 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:58,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42004 deadline: 1680062218433, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:58,439 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:58,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41942 deadline: 1680062218437, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:58,439 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:58,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42024 deadline: 1680062218437, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:58,445 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:58,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42010 deadline: 1680062218438, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:58,543 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:58,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42004 deadline: 1680062218540, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:58,543 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:58,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42024 deadline: 1680062218540, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:58,543 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:58,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41942 deadline: 1680062218540, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:58,549 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:58,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42010 deadline: 1680062218546, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:58,749 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:58,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42004 deadline: 1680062218745, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:58,753 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:58,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42024 deadline: 1680062218746, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:58,753 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:58,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41942 deadline: 1680062218746, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:58,756 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:58,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42010 deadline: 1680062218753, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:58,758 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=411 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/C/015c0e30e7ef4d1b9f79f32b1abca9f5 2023-03-29 03:55:58,765 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/A/7141a92d45374ef8920636399cf79454 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/7141a92d45374ef8920636399cf79454 2023-03-29 03:55:58,770 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/7141a92d45374ef8920636399cf79454, entries=150, sequenceid=411, filesize=11.9 K 2023-03-29 03:55:58,771 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/B/40f05e34f4cf4856930507d82699303b as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/40f05e34f4cf4856930507d82699303b 2023-03-29 03:55:58,776 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/40f05e34f4cf4856930507d82699303b, entries=150, sequenceid=411, filesize=11.9 K 2023-03-29 03:55:58,776 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/C/015c0e30e7ef4d1b9f79f32b1abca9f5 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/015c0e30e7ef4d1b9f79f32b1abca9f5 2023-03-29 03:55:58,779 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/015c0e30e7ef4d1b9f79f32b1abca9f5, entries=150, sequenceid=411, filesize=11.9 K 2023-03-29 03:55:58,780 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 0337be0f939be2a2b593eefecee32758 in 486ms, sequenceid=411, compaction requested=true 2023-03-29 03:55:58,780 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0337be0f939be2a2b593eefecee32758: 2023-03-29 03:55:58,780 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:58,780 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:58,780 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:58,780 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:58,780 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:58,780 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:58,788 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:58,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:58,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:58,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:58,788 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:58,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:58,789 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:58,789 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:58,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:58,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:58,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:58,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:58,792 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:58,792 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@c80b78c[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:58,792 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:58,792 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:58,793 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:58,793 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:58,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:58,793 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:58,793 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:58,793 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:58,794 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:58,794 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:58,795 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:58,795 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:58,795 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:58,795 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:58,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:58,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:58,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:58,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:58,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:58,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:58,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:58,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:58,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:58,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:58,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:58,802 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:58,802 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:58,802 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:58,802 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:58,802 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:58,802 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:58,805 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:58,805 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:58,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:58,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:58,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:58,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:58,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:58,805 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:58,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:58,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:58,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:58,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:58,806 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:58,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:58,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:58,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:58,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:58,810 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:58,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:58,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:58,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:58,813 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:58,813 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:58,813 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:58,813 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:58,813 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:58,813 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:58,813 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:58,813 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:58,814 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:58,814 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:58,814 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:58,814 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:58,814 DEBUG [Thread-3261] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:58,814 DEBUG [Thread-3261] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:58,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:58,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:58,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:58,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:58,914 DEBUG [Thread-3261] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:58,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:58,919 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:58,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:58,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:58,932 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:58,932 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:58,932 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:58,932 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:58,932 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:58,933 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:58,934 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:58,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:58,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:58,934 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:58,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:58,935 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:58,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:58,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:58,935 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:58,935 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:58,935 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:58,940 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:58,940 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:58,940 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:58,940 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:58,940 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:58,948 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:58,948 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:58,948 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:58,948 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. started... 2023-03-29 03:55:58,948 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0337be0f939be2a2b593eefecee32758 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-03-29 03:55:58,948 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0337be0f939be2a2b593eefecee32758, store=A 2023-03-29 03:55:58,949 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:55:58,949 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0337be0f939be2a2b593eefecee32758, store=B 2023-03-29 03:55:58,949 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:55:58,949 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0337be0f939be2a2b593eefecee32758, store=C 2023-03-29 03:55:58,949 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:55:58,983 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=427 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/A/51f8501cb4754894a9f1e65d7ddfafd1 2023-03-29 03:55:59,026 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=427 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/B/108861cc9bf849a58c9072eabc5962de 2023-03-29 03:55:59,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on 0337be0f939be2a2b593eefecee32758 2023-03-29 03:55:59,061 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. as already flushing 2023-03-29 03:55:59,066 WARN [DataStreamer for file /user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/C/cf0b02467c05415381ca134a3e7a61e8] hdfs.DataStreamer(982): Caught exception java.lang.InterruptedException at java.lang.Object.wait(Native Method) at java.lang.Thread.join(Thread.java:1257) at java.lang.Thread.join(Thread.java:1331) at org.apache.hadoop.hdfs.DataStreamer.closeResponder(DataStreamer.java:980) at org.apache.hadoop.hdfs.DataStreamer.endBlock(DataStreamer.java:630) at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:807) 2023-03-29 03:55:59,068 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=427 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/C/cf0b02467c05415381ca134a3e7a61e8 2023-03-29 03:55:59,073 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/A/51f8501cb4754894a9f1e65d7ddfafd1 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/51f8501cb4754894a9f1e65d7ddfafd1 2023-03-29 03:55:59,080 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/51f8501cb4754894a9f1e65d7ddfafd1, entries=150, sequenceid=427, filesize=11.9 K 2023-03-29 03:55:59,081 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/B/108861cc9bf849a58c9072eabc5962de as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/108861cc9bf849a58c9072eabc5962de 2023-03-29 03:55:59,088 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/108861cc9bf849a58c9072eabc5962de, entries=150, sequenceid=427, filesize=11.9 K 2023-03-29 03:55:59,089 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/C/cf0b02467c05415381ca134a3e7a61e8 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/cf0b02467c05415381ca134a3e7a61e8 2023-03-29 03:55:59,093 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/cf0b02467c05415381ca134a3e7a61e8, entries=150, sequenceid=427, filesize=11.9 K 2023-03-29 03:55:59,094 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=67.09 KB/68700 for 0337be0f939be2a2b593eefecee32758 in 146ms, sequenceid=427, compaction requested=true 2023-03-29 03:55:59,094 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0337be0f939be2a2b593eefecee32758: 2023-03-29 03:55:59,094 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:59,094 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:59,094 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:59,095 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:59,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on 0337be0f939be2a2b593eefecee32758 2023-03-29 03:55:59,095 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 0337be0f939be2a2b593eefecee32758 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-03-29 03:55:59,095 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0337be0f939be2a2b593eefecee32758, store=A 2023-03-29 03:55:59,095 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:55:59,096 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0337be0f939be2a2b593eefecee32758, store=B 2023-03-29 03:55:59,096 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:55:59,096 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0337be0f939be2a2b593eefecee32758, store=C 2023-03-29 03:55:59,096 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:55:59,100 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:59,100 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:59,111 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:59,111 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:59,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:59,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:59,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:59,112 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:59,112 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:59,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:59,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:59,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:59,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:59,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:59,114 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:59,114 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@63585c54[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:59,114 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:59,114 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:59,115 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:59,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:59,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:59,116 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:59,116 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:59,116 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:59,116 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:59,116 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:59,119 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:59,119 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:59,119 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:59,120 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:59,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:59,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:59,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:59,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:59,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:59,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:59,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:59,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:59,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:59,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:59,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:59,122 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:59,122 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:59,122 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:59,122 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:59,122 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:59,122 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:59,125 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:59,125 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:59,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:59,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:59,125 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:59,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:59,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:59,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:59,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:59,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:59,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:59,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:59,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:59,126 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:59,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:59,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:59,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:59,127 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:59,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:59,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:59,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:59,130 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:59,130 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:59,130 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:59,130 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:59,130 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:59,130 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:59,130 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:59,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:59,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:59,130 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:59,131 DEBUG [Thread-3261] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:59,131 DEBUG [Thread-3261] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:59,130 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:59,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:59,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:59,130 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:59,131 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:59,131 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:59,134 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=442 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/A/f42a3d7080724d9f990f9d27dc19e9ac 2023-03-29 03:55:59,152 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:59,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42010 deadline: 1680062219144, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:59,164 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:59,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 250 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42004 deadline: 1680062219152, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:59,164 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:59,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42024 deadline: 1680062219154, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:59,167 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:59,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41942 deadline: 1680062219156, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:59,174 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=442 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/B/90d4c81ecf2346cc90aab26a958a6d0b 2023-03-29 03:55:59,197 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=442 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/C/060c4b6e41544963b446beb0ef6415e8 2023-03-29 03:55:59,200 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/A/f42a3d7080724d9f990f9d27dc19e9ac as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/f42a3d7080724d9f990f9d27dc19e9ac 2023-03-29 03:55:59,205 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/f42a3d7080724d9f990f9d27dc19e9ac, entries=400, sequenceid=442, filesize=23.8 K 2023-03-29 03:55:59,206 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/B/90d4c81ecf2346cc90aab26a958a6d0b as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/90d4c81ecf2346cc90aab26a958a6d0b 2023-03-29 03:55:59,212 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/90d4c81ecf2346cc90aab26a958a6d0b, entries=150, sequenceid=442, filesize=11.9 K 2023-03-29 03:55:59,212 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/C/060c4b6e41544963b446beb0ef6415e8 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/060c4b6e41544963b446beb0ef6415e8 2023-03-29 03:55:59,225 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/060c4b6e41544963b446beb0ef6415e8, entries=150, sequenceid=442, filesize=11.9 K 2023-03-29 03:55:59,226 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=100.63 KB/103050 for 0337be0f939be2a2b593eefecee32758 in 131ms, sequenceid=442, compaction requested=true 2023-03-29 03:55:59,226 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 0337be0f939be2a2b593eefecee32758: 2023-03-29 03:55:59,226 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-29 03:55:59,226 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:55:59,227 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:55:59,227 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:55:59,227 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-29 03:55:59,228 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 76494 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-29 03:55:59,228 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1912): 0337be0f939be2a2b593eefecee32758/A is initiating minor compaction (all files) 2023-03-29 03:55:59,228 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 0337be0f939be2a2b593eefecee32758/A in TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:59,228 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/dfe69b2dd25d4dd0b91387ae0e2d41e4, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/4db6ee64cb5a4af4a1e13f51ebe3176f, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/7141a92d45374ef8920636399cf79454, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/51f8501cb4754894a9f1e65d7ddfafd1, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/f42a3d7080724d9f990f9d27dc19e9ac] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp, totalSize=74.7 K 2023-03-29 03:55:59,228 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting dfe69b2dd25d4dd0b91387ae0e2d41e4, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=376, earliestPutTs=1680062157058 2023-03-29 03:55:59,229 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 4db6ee64cb5a4af4a1e13f51ebe3176f, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=392, earliestPutTs=1680062157100 2023-03-29 03:55:59,229 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 7141a92d45374ef8920636399cf79454, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=411, earliestPutTs=1680062157269 2023-03-29 03:55:59,229 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61850 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-29 03:55:59,229 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1912): 0337be0f939be2a2b593eefecee32758/B is initiating minor compaction (all files) 2023-03-29 03:55:59,229 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0337be0f939be2a2b593eefecee32758/B in TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:59,230 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/844eafdd81e846a7a821b9853a7a670c, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/034bb61da4d740b28952a27021b0526c, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/40f05e34f4cf4856930507d82699303b, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/108861cc9bf849a58c9072eabc5962de, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/90d4c81ecf2346cc90aab26a958a6d0b] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp, totalSize=60.4 K 2023-03-29 03:55:59,230 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 51f8501cb4754894a9f1e65d7ddfafd1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=427, earliestPutTs=1680062158432 2023-03-29 03:55:59,230 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 844eafdd81e846a7a821b9853a7a670c, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=376, earliestPutTs=1680062157058 2023-03-29 03:55:59,230 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting f42a3d7080724d9f990f9d27dc19e9ac, keycount=400, bloomtype=ROW, size=23.8 K, encoding=NONE, compression=NONE, seqNum=442, earliestPutTs=1680062159064 2023-03-29 03:55:59,231 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 034bb61da4d740b28952a27021b0526c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=392, earliestPutTs=1680062157100 2023-03-29 03:55:59,231 DEBUG [Thread-3261] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:59,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:59,232 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 40f05e34f4cf4856930507d82699303b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=411, earliestPutTs=1680062157269 2023-03-29 03:55:59,232 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 108861cc9bf849a58c9072eabc5962de, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=427, earliestPutTs=1680062158432 2023-03-29 03:55:59,232 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 90d4c81ecf2346cc90aab26a958a6d0b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=442, earliestPutTs=1680062159077 2023-03-29 03:55:59,241 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:59,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:59,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:59,244 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:59,244 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:59,244 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:59,244 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:59,244 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:59,244 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:59,245 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:59,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:59,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:59,246 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:59,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:59,246 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:59,246 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:59,246 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:59,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:59,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:59,247 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:59,256 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:59,256 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:59,256 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:59,256 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:59,256 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:59,261 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0337be0f939be2a2b593eefecee32758#A#compaction#523 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:55:59,264 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 0337be0f939be2a2b593eefecee32758 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-29 03:55:59,264 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0337be0f939be2a2b593eefecee32758, store=A 2023-03-29 03:55:59,264 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:55:59,264 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0337be0f939be2a2b593eefecee32758, store=B 2023-03-29 03:55:59,264 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:55:59,264 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0337be0f939be2a2b593eefecee32758, store=C 2023-03-29 03:55:59,264 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:55:59,264 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] throttle.PressureAwareThroughputController(145): 0337be0f939be2a2b593eefecee32758#B#compaction#524 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:55:59,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on 0337be0f939be2a2b593eefecee32758 2023-03-29 03:55:59,268 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:59,268 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:59,268 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:59,268 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. started... 2023-03-29 03:55:59,268 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. as already flushing 2023-03-29 03:55:59,268 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:59,268 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:59,268 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:59,268 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:59,289 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:59,290 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:59,295 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:59,296 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:59,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:59,296 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:59,296 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:59,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:59,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:59,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:59,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:59,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:59,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:59,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:59,305 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:59,305 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@646f8b62[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:59,305 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:59,305 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:59,307 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:59,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:59,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:59,307 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:59,307 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:59,307 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:59,308 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:59,308 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:59,309 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:59,309 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:59,309 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:59,309 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:59,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:59,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:59,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:59,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:59,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:59,310 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/B/534d4694ad7d438397e2812bc07d748e as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/534d4694ad7d438397e2812bc07d748e 2023-03-29 03:55:59,310 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/A/c0f72296859d4937be1da21f5d2ce54d as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/c0f72296859d4937be1da21f5d2ce54d 2023-03-29 03:55:59,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:59,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:59,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:59,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:59,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:59,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:59,312 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:59,312 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:59,312 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:59,312 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:59,313 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:59,313 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:59,314 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:59,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41942 deadline: 1680062219296, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:59,315 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:59,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 254 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42004 deadline: 1680062219296, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:59,315 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:59,315 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:59,315 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:59,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:59,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:59,315 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:59,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:59,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:59,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42010 deadline: 1680062219297, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:59,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:59,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:59,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:59,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:59,317 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 0337be0f939be2a2b593eefecee32758/A of 0337be0f939be2a2b593eefecee32758 into c0f72296859d4937be1da21f5d2ce54d(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:55:59,317 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0337be0f939be2a2b593eefecee32758: 2023-03-29 03:55:59,317 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758., storeName=0337be0f939be2a2b593eefecee32758/A, priority=11, startTime=1680062159226; duration=0sec 2023-03-29 03:55:59,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:59,317 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:55:59,317 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-29 03:55:59,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:59,317 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:59,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:59,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:59,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:59,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:59,318 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:59,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:59,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:59,320 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61850 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-29 03:55:59,320 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1912): 0337be0f939be2a2b593eefecee32758/C is initiating minor compaction (all files) 2023-03-29 03:55:59,320 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 0337be0f939be2a2b593eefecee32758/C in TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:59,320 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/ae7f434d973b4520819f2fed2e18375b, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/b2d96e8382364215ba9580656d165323, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/015c0e30e7ef4d1b9f79f32b1abca9f5, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/cf0b02467c05415381ca134a3e7a61e8, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/060c4b6e41544963b446beb0ef6415e8] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp, totalSize=60.4 K 2023-03-29 03:55:59,326 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:59,326 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:59,326 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:59,326 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:59,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:59,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:59,326 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:59,326 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:59,326 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:59,326 DEBUG [Thread-3261] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:59,326 DEBUG [Thread-3261] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:59,326 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting ae7f434d973b4520819f2fed2e18375b, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=376, earliestPutTs=1680062157058 2023-03-29 03:55:59,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:59,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:59,326 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:59,327 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting b2d96e8382364215ba9580656d165323, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=392, earliestPutTs=1680062157100 2023-03-29 03:55:59,327 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:59,327 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:59,327 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:59,327 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:59,327 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 015c0e30e7ef4d1b9f79f32b1abca9f5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=411, earliestPutTs=1680062157269 2023-03-29 03:55:59,327 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting cf0b02467c05415381ca134a3e7a61e8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=427, earliestPutTs=1680062158432 2023-03-29 03:55:59,328 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:59,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42024 deadline: 1680062219314, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:59,328 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 060c4b6e41544963b446beb0ef6415e8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=442, earliestPutTs=1680062159077 2023-03-29 03:55:59,334 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 0337be0f939be2a2b593eefecee32758/B of 0337be0f939be2a2b593eefecee32758 into 534d4694ad7d438397e2812bc07d748e(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:55:59,334 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0337be0f939be2a2b593eefecee32758: 2023-03-29 03:55:59,334 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758., storeName=0337be0f939be2a2b593eefecee32758/B, priority=11, startTime=1680062159227; duration=0sec 2023-03-29 03:55:59,335 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:55:59,360 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0337be0f939be2a2b593eefecee32758#C#compaction#526 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:55:59,402 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/C/18afc47121e34a40bf5e67294505ff48 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/18afc47121e34a40bf5e67294505ff48 2023-03-29 03:55:59,406 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 0337be0f939be2a2b593eefecee32758/C of 0337be0f939be2a2b593eefecee32758 into 18afc47121e34a40bf5e67294505ff48(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:55:59,407 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0337be0f939be2a2b593eefecee32758: 2023-03-29 03:55:59,407 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758., storeName=0337be0f939be2a2b593eefecee32758/C, priority=11, startTime=1680062159227; duration=0sec 2023-03-29 03:55:59,407 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:55:59,420 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:59,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41942 deadline: 1680062219415, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:59,420 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:59,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 256 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42004 deadline: 1680062219416, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:59,421 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:59,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42010 deadline: 1680062219417, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:59,426 DEBUG [Thread-3261] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:59,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:59,429 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:59,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:59,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:59,432 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:59,432 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:59,432 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:59,432 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:59,433 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:59,433 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:59,435 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:59,435 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:59,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42024 deadline: 1680062219429, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:59,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:59,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:59,437 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:59,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:59,437 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:59,437 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:59,437 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:59,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:59,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:59,439 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:59,440 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:59,440 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:59,440 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:59,440 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:59,440 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:59,444 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:59,444 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:59,444 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:59,444 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. started... 2023-03-29 03:55:59,444 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. as already flushing 2023-03-29 03:55:59,444 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:59,444 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:59,444 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:59,444 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:59,444 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:59,444 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:59,449 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:59,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:59,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:59,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:59,449 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:59,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:59,449 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:59,449 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:59,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:59,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:59,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:59,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:59,451 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:59,451 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4607761a[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:59,451 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:59,451 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:59,455 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:59,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:59,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:59,456 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:59,456 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:59,456 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:59,456 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:59,456 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:59,459 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:59,459 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:59,459 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:59,459 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:59,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:59,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:59,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:59,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:59,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:59,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:59,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:59,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:59,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:59,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:59,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:59,464 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:59,464 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:59,464 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:59,464 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:59,464 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:59,464 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:59,466 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:59,466 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:59,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:59,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:59,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:59,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:59,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:59,466 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:59,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:59,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:59,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:59,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:59,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:59,468 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:59,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:59,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:59,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:59,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:59,473 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:59,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:59,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:59,475 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:59,475 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:59,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:59,476 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:59,476 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:59,476 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:59,476 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:59,475 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:59,476 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:59,476 DEBUG [Thread-3261] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:59,476 DEBUG [Thread-3261] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:59,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:59,476 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:59,476 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:59,476 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:59,476 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:59,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:59,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:59,576 DEBUG [Thread-3261] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:59,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:59,579 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:59,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:59,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:59,587 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:59,587 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:59,587 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:59,587 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:59,588 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:59,588 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:59,589 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:59,589 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:59,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:59,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:59,591 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:59,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:59,591 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:59,592 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:59,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:59,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:59,592 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:59,604 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:59,604 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:59,604 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:59,604 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:59,604 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:59,620 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:59,620 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:59,620 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:59,620 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. started... 2023-03-29 03:55:59,620 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. as already flushing 2023-03-29 03:55:59,620 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:59,620 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:59,620 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:59,620 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:59,620 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:59,620 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:59,623 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:59,623 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:59,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:59,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:59,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:59,623 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:59,623 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:59,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:59,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:59,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:59,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:59,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:59,625 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:59,625 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4fb29581[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:59,625 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:59,625 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:59,626 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:59,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:59,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:59,626 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:59,627 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:59,627 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:59,627 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:59,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42010 deadline: 1680062219622, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:59,627 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:59,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41942 deadline: 1680062219624, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:59,627 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:59,628 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:59,628 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:59,628 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:59,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:59,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:59,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:59,628 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:59,628 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:59,628 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:59,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 258 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42004 deadline: 1680062219623, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:59,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:59,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:59,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:59,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:59,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:59,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:59,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:59,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:59,630 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:59,631 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:59,631 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:59,631 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:59,631 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:59,631 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:59,632 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:59,632 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:59,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:59,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:59,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:59,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:59,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:59,632 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:59,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:59,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:59,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:59,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:59,633 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:59,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:59,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:59,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:59,634 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:59,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:59,637 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:59,637 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:59,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:59,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:59,637 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:59,637 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:59,637 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:59,637 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:59,637 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:59,637 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:59,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:59,637 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:59,637 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:59,637 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:59,638 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:59,638 DEBUG [Thread-3261] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:59,638 DEBUG [Thread-3261] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:59,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:59,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:59,638 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:55:59,641 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:59,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42024 deadline: 1680062219638, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:59,715 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=461 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/A/d4165a46320a4f998fe47c5dc08a2bd0 2023-03-29 03:55:59,738 DEBUG [Thread-3261] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:59,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:59,740 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:59,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:59,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:59,744 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:59,744 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:59,744 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:59,744 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:59,744 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:59,745 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:59,746 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=461 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/B/39c4248edeb44cf889f63c1f1e3f3417 2023-03-29 03:55:59,746 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:59,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:59,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:59,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:59,747 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:59,747 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:59,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:59,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:59,748 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:59,748 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:59,748 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:59,748 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:59,748 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:59,748 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:59,748 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:59,748 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:59,748 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:59,748 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:59,748 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:59,748 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. started... 2023-03-29 03:55:59,748 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. as already flushing 2023-03-29 03:55:59,749 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:59,749 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:55:59,749 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:55:59,749 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:59,749 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:55:59,749 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:55:59,751 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:59,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:59,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:59,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:59,751 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:59,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:59,751 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:59,752 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:55:59,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:59,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:59,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:59,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:59,759 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:55:59,759 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ea9cadf[Count = 0] remaining members to acquire global barrier 2023-03-29 03:55:59,759 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:55:59,759 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:59,762 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:59,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:59,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:59,763 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:55:59,763 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:55:59,763 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:55:59,763 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:59,763 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:55:59,765 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:55:59,765 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:59,765 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:55:59,765 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:59,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:59,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:59,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:59,766 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=461 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/C/d6eba1304e794028a00feae7ceece564 2023-03-29 03:55:59,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:59,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:59,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:59,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:59,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:59,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:59,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:59,772 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/A/d4165a46320a4f998fe47c5dc08a2bd0 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/d4165a46320a4f998fe47c5dc08a2bd0 2023-03-29 03:55:59,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:55:59,772 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:55:59,772 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:55:59,772 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:55:59,773 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:55:59,773 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:55:59,773 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:55:59,777 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:59,778 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:59,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:59,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:55:59,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:55:59,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:59,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:59,778 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:59,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:59,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:59,778 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:59,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:55:59,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:59,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:59,779 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/d4165a46320a4f998fe47c5dc08a2bd0, entries=200, sequenceid=461, filesize=14.3 K 2023-03-29 03:55:59,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:55:59,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:59,780 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:59,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:59,781 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/B/39c4248edeb44cf889f63c1f1e3f3417 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/39c4248edeb44cf889f63c1f1e3f3417 2023-03-29 03:55:59,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:55:59,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:55:59,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:59,785 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/39c4248edeb44cf889f63c1f1e3f3417, entries=150, sequenceid=461, filesize=11.9 K 2023-03-29 03:55:59,786 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/C/d6eba1304e794028a00feae7ceece564 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/d6eba1304e794028a00feae7ceece564 2023-03-29 03:55:59,790 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:59,790 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:55:59,790 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:59,790 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:59,790 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:59,790 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:59,790 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:59,790 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:55:59,790 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:59,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:59,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:59,791 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:55:59,791 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:55:59,791 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:55:59,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:55:59,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:55:59,792 DEBUG [Thread-3261] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:55:59,792 DEBUG [Thread-3261] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:55:59,793 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/d6eba1304e794028a00feae7ceece564, entries=150, sequenceid=461, filesize=11.9 K 2023-03-29 03:55:59,794 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 0337be0f939be2a2b593eefecee32758 in 531ms, sequenceid=461, compaction requested=false 2023-03-29 03:55:59,794 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 0337be0f939be2a2b593eefecee32758: 2023-03-29 03:55:59,794 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:55:59,892 DEBUG [Thread-3261] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:55:59,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:55:59,894 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:55:59,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:55:59,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:55:59,895 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:55:59,895 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:59,895 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:55:59,895 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:55:59,896 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:59,896 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:59,899 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:59,899 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:55:59,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:55:59,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:55:59,899 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:59,899 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:55:59,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:59,900 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:55:59,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:55:59,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:55:59,901 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:55:59,902 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:55:59,902 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:55:59,902 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:55:59,902 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:55:59,902 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:55:59,904 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:55:59,904 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:55:59,904 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:55:59,904 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. started... 2023-03-29 03:55:59,904 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0337be0f939be2a2b593eefecee32758 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-29 03:55:59,904 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0337be0f939be2a2b593eefecee32758, store=A 2023-03-29 03:55:59,904 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:55:59,904 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0337be0f939be2a2b593eefecee32758, store=B 2023-03-29 03:55:59,905 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:55:59,905 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0337be0f939be2a2b593eefecee32758, store=C 2023-03-29 03:55:59,905 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:55:59,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on 0337be0f939be2a2b593eefecee32758 2023-03-29 03:55:59,933 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. as already flushing 2023-03-29 03:55:59,953 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=477 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/A/613125bcca8045bdbab5d5850fdc9036 2023-03-29 03:55:59,995 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:59,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41942 deadline: 1680062219983, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:59,995 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:59,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 264 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42004 deadline: 1680062219983, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:59,995 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:59,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42024 deadline: 1680062219983, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:55:59,996 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:55:59,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 256 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42010 deadline: 1680062219984, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:00,102 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:00,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41942 deadline: 1680062220096, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:00,103 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:00,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 266 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42004 deadline: 1680062220096, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:00,103 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:00,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42024 deadline: 1680062220096, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:00,103 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:00,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 258 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42010 deadline: 1680062220096, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:00,309 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:00,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 268 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42004 deadline: 1680062220304, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:00,310 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:00,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41942 deadline: 1680062220304, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:00,310 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:00,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 260 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42010 deadline: 1680062220305, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:00,309 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:00,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:42024 deadline: 1680062220304, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:00,375 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=477 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/B/31672a1c8ab14c97a2a0c892081d42f0 2023-03-29 03:56:00,386 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=477 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/C/697b3bcc193d48b1af4772e35c465734 2023-03-29 03:56:00,390 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/A/613125bcca8045bdbab5d5850fdc9036 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/613125bcca8045bdbab5d5850fdc9036 2023-03-29 03:56:00,396 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/613125bcca8045bdbab5d5850fdc9036, entries=150, sequenceid=477, filesize=11.9 K 2023-03-29 03:56:00,397 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/B/31672a1c8ab14c97a2a0c892081d42f0 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/31672a1c8ab14c97a2a0c892081d42f0 2023-03-29 03:56:00,401 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/31672a1c8ab14c97a2a0c892081d42f0, entries=150, sequenceid=477, filesize=11.9 K 2023-03-29 03:56:00,401 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/C/697b3bcc193d48b1af4772e35c465734 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/697b3bcc193d48b1af4772e35c465734 2023-03-29 03:56:00,407 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/697b3bcc193d48b1af4772e35c465734, entries=150, sequenceid=477, filesize=11.9 K 2023-03-29 03:56:00,407 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 0337be0f939be2a2b593eefecee32758 in 503ms, sequenceid=477, compaction requested=true 2023-03-29 03:56:00,407 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0337be0f939be2a2b593eefecee32758: 2023-03-29 03:56:00,408 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:56:00,408 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:56:00,408 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:56:00,408 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:00,408 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:56:00,408 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:56:00,409 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:00,409 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:00,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:00,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:00,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:00,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:00,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:00,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:00,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:00,411 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:00,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:00,411 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:56:00,411 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:56:00,411 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@768568f3[Count = 0] remaining members to acquire global barrier 2023-03-29 03:56:00,411 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:56:00,411 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:00,413 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:00,413 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:56:00,413 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:00,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:00,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:00,413 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:56:00,413 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:56:00,413 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:56:00,415 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:00,415 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:56:00,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:00,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:00,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:00,415 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:00,415 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:56:00,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:00,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:00,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:00,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:00,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:00,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:00,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:00,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:56:00,419 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:56:00,419 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:56:00,419 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:56:00,419 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:56:00,419 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:56:00,419 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:56:00,421 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:00,421 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:00,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:00,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:00,421 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:00,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:00,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:00,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:00,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:00,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:00,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:00,422 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:00,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:00,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:00,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:00,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:00,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:00,423 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:00,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:00,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:00,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:00,425 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:00,425 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:00,426 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:00,426 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:56:00,426 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:56:00,425 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:00,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:00,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:00,426 DEBUG [Thread-3261] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:56:00,426 DEBUG [Thread-3261] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:56:00,426 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:00,426 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:00,426 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:00,426 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:00,426 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:00,426 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:00,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:00,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:00,526 DEBUG [Thread-3261] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:56:00,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:56:00,528 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:56:00,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:56:00,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:56:00,530 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:56:00,530 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:00,530 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:56:00,530 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:56:00,530 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:00,530 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:00,532 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:00,532 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:00,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:00,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:00,532 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:00,532 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:56:00,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:00,533 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:00,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:56:00,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:00,533 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:56:00,533 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:56:00,533 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:56:00,533 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:56:00,534 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:00,534 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:56:00,534 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:56:00,534 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:56:00,534 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:56:00,534 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. started... 2023-03-29 03:56:00,534 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0337be0f939be2a2b593eefecee32758 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-29 03:56:00,534 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0337be0f939be2a2b593eefecee32758, store=A 2023-03-29 03:56:00,535 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:56:00,535 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0337be0f939be2a2b593eefecee32758, store=B 2023-03-29 03:56:00,535 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:56:00,535 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0337be0f939be2a2b593eefecee32758, store=C 2023-03-29 03:56:00,535 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:56:00,547 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=496 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/A/52bc362d09cd4b8a834ff56e0ba0a863 2023-03-29 03:56:00,568 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=496 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/B/04e6e48575f4443887aac3a6f3d7d176 2023-03-29 03:56:00,596 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=496 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/C/a576371842fc41e889197e2cd8db3f4a 2023-03-29 03:56:00,604 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/A/52bc362d09cd4b8a834ff56e0ba0a863 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/52bc362d09cd4b8a834ff56e0ba0a863 2023-03-29 03:56:00,609 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/52bc362d09cd4b8a834ff56e0ba0a863, entries=150, sequenceid=496, filesize=11.9 K 2023-03-29 03:56:00,611 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/B/04e6e48575f4443887aac3a6f3d7d176 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/04e6e48575f4443887aac3a6f3d7d176 2023-03-29 03:56:00,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on 0337be0f939be2a2b593eefecee32758 2023-03-29 03:56:00,617 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. as already flushing 2023-03-29 03:56:00,618 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/04e6e48575f4443887aac3a6f3d7d176, entries=150, sequenceid=496, filesize=11.9 K 2023-03-29 03:56:00,621 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/C/a576371842fc41e889197e2cd8db3f4a as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/a576371842fc41e889197e2cd8db3f4a 2023-03-29 03:56:00,627 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/a576371842fc41e889197e2cd8db3f4a, entries=150, sequenceid=496, filesize=11.9 K 2023-03-29 03:56:00,628 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=40.25 KB/41220 for 0337be0f939be2a2b593eefecee32758 in 94ms, sequenceid=496, compaction requested=true 2023-03-29 03:56:00,628 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0337be0f939be2a2b593eefecee32758: 2023-03-29 03:56:00,628 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:56:00,628 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:56:00,628 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:56:00,628 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:00,628 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:56:00,628 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:56:00,631 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:00,631 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:00,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on 0337be0f939be2a2b593eefecee32758 2023-03-29 03:56:00,631 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 0337be0f939be2a2b593eefecee32758 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-29 03:56:00,631 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:00,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:00,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:00,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:00,631 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:56:00,631 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0337be0f939be2a2b593eefecee32758, store=A 2023-03-29 03:56:00,631 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:56:00,631 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0337be0f939be2a2b593eefecee32758, store=B 2023-03-29 03:56:00,631 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:56:00,632 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0337be0f939be2a2b593eefecee32758, store=C 2023-03-29 03:56:00,632 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:56:00,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:00,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:00,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:00,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:00,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:00,635 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:56:00,635 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b4f779f[Count = 0] remaining members to acquire global barrier 2023-03-29 03:56:00,635 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:56:00,635 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:00,636 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:00,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:00,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:00,637 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:56:00,637 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:56:00,637 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:00,637 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:56:00,637 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:56:00,639 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:00,639 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:56:00,639 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:00,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:00,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:00,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:00,639 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:56:00,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:00,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:00,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:00,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:00,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:00,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:00,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:00,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:56:00,641 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:56:00,641 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:56:00,641 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:56:00,641 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:56:00,641 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:56:00,641 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:56:00,643 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:00,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:00,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:00,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:00,644 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:00,644 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:00,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:00,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:00,644 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:00,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:00,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:00,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:00,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:00,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:00,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:00,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:00,647 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:00,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:00,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:00,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:00,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:00,650 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:00,650 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:00,650 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:00,650 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:56:00,650 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:56:00,650 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:00,650 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:00,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:00,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:00,650 DEBUG [Thread-3261] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:56:00,650 DEBUG [Thread-3261] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:56:00,650 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:00,650 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:00,650 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:00,650 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:00,650 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:56:00,650 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:00,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:00,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:00,728 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680062160659, max timestamp=1680062160726Num uniques 859; 2023-03-29 03:56:00,728 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-29 03:56:00,728 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680062160659, max timestamp=1680062160726Num uniques 859; 2023-03-29 03:56:00,728 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680062160659, max timestamp=1680062160726Num uniques 859; 2023-03-29 03:56:00,728 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-29 03:56:00,729 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-29 03:56:00,750 DEBUG [Thread-3261] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:56:00,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:56:00,753 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:56:00,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:56:00,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:56:00,754 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:56:00,754 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:00,754 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:56:00,754 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:56:00,754 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:00,755 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:00,756 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:00,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:00,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:00,756 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:00,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:00,757 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:00,757 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:56:00,757 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:00,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:56:00,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:00,757 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:56:00,757 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:56:00,757 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:56:00,757 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:56:00,757 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:00,758 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:56:00,758 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:56:00,758 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:56:00,758 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:56:00,758 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. started... 2023-03-29 03:56:00,758 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. as already flushing 2023-03-29 03:56:00,758 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:56:00,758 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:56:00,758 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:56:00,758 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:00,758 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:56:00,758 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:56:00,760 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:00,760 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:00,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:00,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:00,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:00,760 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:00,760 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:56:00,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:00,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:00,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:00,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:00,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:00,761 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:56:00,761 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@500af5ea[Count = 0] remaining members to acquire global barrier 2023-03-29 03:56:00,761 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:56:00,761 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:00,763 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:00,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:00,763 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:00,763 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:56:00,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:00,763 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:56:00,763 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:56:00,763 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:56:00,766 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:00,766 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:56:00,766 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:00,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:00,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:00,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:00,766 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:56:00,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:00,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:00,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:00,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:00,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:00,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:00,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:00,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:56:00,769 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:56:00,769 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:56:00,769 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:56:00,769 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:56:00,769 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:56:00,769 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:56:00,771 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:00,771 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:00,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:00,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:00,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:00,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:00,771 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:00,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:00,772 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:00,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:00,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:00,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:00,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:00,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:00,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:00,773 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:00,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:00,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:00,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:00,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:00,775 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:00,776 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:00,776 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:56:00,776 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:56:00,776 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:00,776 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:00,776 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:00,776 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:00,776 DEBUG [Thread-3261] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:56:00,776 DEBUG [Thread-3261] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:56:00,776 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:00,776 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:00,776 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:00,776 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:00,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:00,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:00,776 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:56:00,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:00,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:00,788 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:00,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42024 deadline: 1680062220769, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:00,788 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:00,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 280 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42004 deadline: 1680062220773, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:00,789 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:00,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41942 deadline: 1680062220774, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:00,790 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:00,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 273 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42010 deadline: 1680062220776, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:00,876 DEBUG [Thread-3261] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:56:00,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:56:00,878 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:56:00,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:56:00,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:56:00,879 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:56:00,879 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:00,879 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:56:00,879 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:56:00,880 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:00,880 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:00,885 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:00,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:00,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:00,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:00,886 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:00,886 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:00,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:56:00,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:00,886 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:56:00,886 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:00,887 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:56:00,886 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:56:00,887 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:56:00,887 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:56:00,887 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:00,887 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:56:00,887 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:56:00,887 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:56:00,887 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:56:00,887 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. started... 2023-03-29 03:56:00,887 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. as already flushing 2023-03-29 03:56:00,887 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:56:00,887 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:56:00,887 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:56:00,887 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:00,887 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:56:00,887 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:56:00,890 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:00,890 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:00,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:00,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:00,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:00,891 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:00,891 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:56:00,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:00,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:00,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:00,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:00,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:00,892 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:56:00,892 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70d53206[Count = 0] remaining members to acquire global barrier 2023-03-29 03:56:00,892 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:56:00,892 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:00,894 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:00,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:00,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:00,894 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:56:00,894 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:56:00,894 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:56:00,896 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:56:00,896 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:00,896 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:56:00,896 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:00,896 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:56:00,896 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:56:00,897 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:56:00,897 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:56:00,897 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:56:00,897 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:56:00,897 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:56:00,896 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:00,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 282 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42004 deadline: 1680062220890, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:00,897 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:00,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42024 deadline: 1680062220890, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:00,897 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:00,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41942 deadline: 1680062220890, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:00,899 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:00,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 275 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42010 deadline: 1680062220891, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:00,900 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:00,900 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:00,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:00,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:00,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:00,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:00,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:00,900 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:00,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:00,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:00,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:00,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:00,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:00,901 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:00,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:00,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:00,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:00,902 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:00,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:00,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:00,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:00,905 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:00,905 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:00,906 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:00,906 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:00,906 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:00,906 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:00,906 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:00,906 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:00,906 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:00,906 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:56:00,906 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:56:00,906 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:00,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:00,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:00,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:00,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:00,907 DEBUG [Thread-3261] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:56:00,907 DEBUG [Thread-3261] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:56:01,007 DEBUG [Thread-3261] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:56:01,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:56:01,009 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:56:01,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:56:01,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:56:01,010 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:56:01,010 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:01,010 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:56:01,011 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:56:01,011 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:01,011 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:01,012 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:01,012 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:01,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:01,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:01,013 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:01,013 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:56:01,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:01,013 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:01,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:56:01,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:01,013 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:56:01,014 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:56:01,014 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:56:01,014 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:56:01,014 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:01,014 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:56:01,014 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:56:01,014 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:56:01,014 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:56:01,014 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. started... 2023-03-29 03:56:01,014 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. as already flushing 2023-03-29 03:56:01,014 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:56:01,014 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:56:01,015 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:56:01,015 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:01,015 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:56:01,015 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:56:01,017 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:01,017 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:01,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:01,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:01,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:01,018 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:01,018 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:56:01,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:01,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:01,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:01,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:01,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:01,019 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:56:01,019 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d4a0e1f[Count = 0] remaining members to acquire global barrier 2023-03-29 03:56:01,019 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:56:01,019 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:01,022 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:01,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:01,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:01,022 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:56:01,022 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:56:01,022 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:56:01,022 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:01,022 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:56:01,026 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:01,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:01,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:01,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:01,026 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:56:01,027 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:01,027 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:56:01,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:01,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:01,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:01,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:01,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:01,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:01,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:01,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:56:01,029 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:56:01,029 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:56:01,029 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:56:01,029 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:56:01,029 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:56:01,029 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:56:01,032 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:01,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:01,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:01,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:01,032 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:01,032 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:01,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:01,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:01,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:01,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:01,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:01,033 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:01,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:01,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:01,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:01,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:01,034 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:01,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:01,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:01,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:01,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:01,037 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:01,037 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:01,037 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:01,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:01,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:01,037 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:01,037 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:01,037 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:01,037 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:01,037 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:01,037 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:01,037 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:56:01,037 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:56:01,037 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:56:01,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:01,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:01,037 DEBUG [Thread-3261] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:56:01,037 DEBUG [Thread-3261] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:56:01,037 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:01,050 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=506 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/A/6ea17ae149da4fc8914a8e7b2455fe92 2023-03-29 03:56:01,062 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=506 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/B/9ad52d585ccd4d32b332da3e0a92a056 2023-03-29 03:56:01,074 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=506 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/C/43d18a2585fb40c4bf0ac76ef9222924 2023-03-29 03:56:01,078 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/A/6ea17ae149da4fc8914a8e7b2455fe92 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/6ea17ae149da4fc8914a8e7b2455fe92 2023-03-29 03:56:01,081 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/6ea17ae149da4fc8914a8e7b2455fe92, entries=150, sequenceid=506, filesize=11.9 K 2023-03-29 03:56:01,082 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/B/9ad52d585ccd4d32b332da3e0a92a056 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/9ad52d585ccd4d32b332da3e0a92a056 2023-03-29 03:56:01,094 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/9ad52d585ccd4d32b332da3e0a92a056, entries=150, sequenceid=506, filesize=11.9 K 2023-03-29 03:56:01,095 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/C/43d18a2585fb40c4bf0ac76ef9222924 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/43d18a2585fb40c4bf0ac76ef9222924 2023-03-29 03:56:01,099 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/43d18a2585fb40c4bf0ac76ef9222924, entries=150, sequenceid=506, filesize=11.9 K 2023-03-29 03:56:01,100 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 0337be0f939be2a2b593eefecee32758 in 469ms, sequenceid=506, compaction requested=true 2023-03-29 03:56:01,100 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 0337be0f939be2a2b593eefecee32758: 2023-03-29 03:56:01,100 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:56:01,100 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-29 03:56:01,100 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:56:01,100 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-29 03:56:01,101 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:56:01,102 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 64460 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-29 03:56:01,102 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 62020 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-29 03:56:01,102 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1912): 0337be0f939be2a2b593eefecee32758/A is initiating minor compaction (all files) 2023-03-29 03:56:01,102 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1912): 0337be0f939be2a2b593eefecee32758/B is initiating minor compaction (all files) 2023-03-29 03:56:01,102 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 0337be0f939be2a2b593eefecee32758/A in TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:56:01,102 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0337be0f939be2a2b593eefecee32758/B in TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:56:01,102 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/c0f72296859d4937be1da21f5d2ce54d, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/d4165a46320a4f998fe47c5dc08a2bd0, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/613125bcca8045bdbab5d5850fdc9036, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/52bc362d09cd4b8a834ff56e0ba0a863, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/6ea17ae149da4fc8914a8e7b2455fe92] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp, totalSize=62.9 K 2023-03-29 03:56:01,102 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/534d4694ad7d438397e2812bc07d748e, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/39c4248edeb44cf889f63c1f1e3f3417, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/31672a1c8ab14c97a2a0c892081d42f0, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/04e6e48575f4443887aac3a6f3d7d176, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/9ad52d585ccd4d32b332da3e0a92a056] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp, totalSize=60.6 K 2023-03-29 03:56:01,102 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting c0f72296859d4937be1da21f5d2ce54d, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=442, earliestPutTs=1680062159077 2023-03-29 03:56:01,103 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 534d4694ad7d438397e2812bc07d748e, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=442, earliestPutTs=1680062159077 2023-03-29 03:56:01,103 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting d4165a46320a4f998fe47c5dc08a2bd0, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=461, earliestPutTs=1680062159144 2023-03-29 03:56:01,103 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 39c4248edeb44cf889f63c1f1e3f3417, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=461, earliestPutTs=1680062159144 2023-03-29 03:56:01,103 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 613125bcca8045bdbab5d5850fdc9036, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=477, earliestPutTs=1680062159280 2023-03-29 03:56:01,103 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 31672a1c8ab14c97a2a0c892081d42f0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=477, earliestPutTs=1680062159280 2023-03-29 03:56:01,103 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 52bc362d09cd4b8a834ff56e0ba0a863, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=496, earliestPutTs=1680062159955 2023-03-29 03:56:01,103 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 04e6e48575f4443887aac3a6f3d7d176, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=496, earliestPutTs=1680062159955 2023-03-29 03:56:01,104 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 6ea17ae149da4fc8914a8e7b2455fe92, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=506, earliestPutTs=1680062160622 2023-03-29 03:56:01,104 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 9ad52d585ccd4d32b332da3e0a92a056, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=506, earliestPutTs=1680062160622 2023-03-29 03:56:01,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on 0337be0f939be2a2b593eefecee32758 2023-03-29 03:56:01,110 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 0337be0f939be2a2b593eefecee32758 3/3 column families, dataSize=140.89 KB heapSize=405.61 KB 2023-03-29 03:56:01,110 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0337be0f939be2a2b593eefecee32758, store=A 2023-03-29 03:56:01,111 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-29 03:56:01,111 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0337be0f939be2a2b593eefecee32758, store=B 2023-03-29 03:56:01,111 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-29 03:56:01,111 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0337be0f939be2a2b593eefecee32758, store=C 2023-03-29 03:56:01,111 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-29 03:56:01,123 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] throttle.PressureAwareThroughputController(145): 0337be0f939be2a2b593eefecee32758#B#compaction#539 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:56:01,125 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0337be0f939be2a2b593eefecee32758#A#compaction#540 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:56:01,138 DEBUG [Thread-3261] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:56:01,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:56:01,140 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:56:01,141 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:01,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 286 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42004 deadline: 1680062221130, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:01,142 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:01,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41942 deadline: 1680062221131, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:01,142 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:01,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42024 deadline: 1680062221132, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:01,142 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:01,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 279 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42010 deadline: 1680062221132, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:01,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:56:01,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:56:01,151 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:56:01,151 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:01,151 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:56:01,151 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:56:01,151 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:01,151 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:01,154 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:01,154 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:01,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:01,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:01,155 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:01,155 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:56:01,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:01,155 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:01,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:56:01,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:01,156 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:56:01,156 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:56:01,156 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:56:01,156 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:56:01,156 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:01,156 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:56:01,158 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/B/668d41bbbf754f38883ac62f93892179 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/668d41bbbf754f38883ac62f93892179 2023-03-29 03:56:01,160 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:56:01,160 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:56:01,160 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:56:01,160 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. started... 2023-03-29 03:56:01,160 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. as already flushing 2023-03-29 03:56:01,160 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:56:01,160 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:56:01,160 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:56:01,160 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:01,160 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:56:01,160 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:56:01,162 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:01,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:01,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:01,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:01,162 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:01,162 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 0337be0f939be2a2b593eefecee32758/B of 0337be0f939be2a2b593eefecee32758 into 668d41bbbf754f38883ac62f93892179(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:56:01,163 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0337be0f939be2a2b593eefecee32758: 2023-03-29 03:56:01,163 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758., storeName=0337be0f939be2a2b593eefecee32758/B, priority=11, startTime=1680062161100; duration=0sec 2023-03-29 03:56:01,163 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:56:01,163 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-29 03:56:01,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:01,163 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:01,163 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:56:01,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:01,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:01,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:01,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:01,166 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 62020 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-29 03:56:01,166 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1912): 0337be0f939be2a2b593eefecee32758/C is initiating minor compaction (all files) 2023-03-29 03:56:01,166 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0337be0f939be2a2b593eefecee32758/C in TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:56:01,166 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/18afc47121e34a40bf5e67294505ff48, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/d6eba1304e794028a00feae7ceece564, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/697b3bcc193d48b1af4772e35c465734, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/a576371842fc41e889197e2cd8db3f4a, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/43d18a2585fb40c4bf0ac76ef9222924] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp, totalSize=60.6 K 2023-03-29 03:56:01,166 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 18afc47121e34a40bf5e67294505ff48, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=442, earliestPutTs=1680062159077 2023-03-29 03:56:01,166 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:56:01,167 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5dc977a6[Count = 0] remaining members to acquire global barrier 2023-03-29 03:56:01,167 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:56:01,167 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:01,167 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting d6eba1304e794028a00feae7ceece564, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=461, earliestPutTs=1680062159144 2023-03-29 03:56:01,167 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 697b3bcc193d48b1af4772e35c465734, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=477, earliestPutTs=1680062159280 2023-03-29 03:56:01,167 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting a576371842fc41e889197e2cd8db3f4a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=496, earliestPutTs=1680062159955 2023-03-29 03:56:01,168 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 43d18a2585fb40c4bf0ac76ef9222924, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=506, earliestPutTs=1680062160622 2023-03-29 03:56:01,168 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:01,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:01,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:01,168 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:56:01,168 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:56:01,168 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:56:01,168 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:01,168 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:56:01,170 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:56:01,170 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:01,170 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:01,170 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:56:01,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:01,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:01,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:01,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:01,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:01,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:01,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:01,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:01,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:01,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:01,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:56:01,172 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:56:01,173 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:56:01,173 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:56:01,173 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:56:01,173 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:56:01,173 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:56:01,175 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:01,175 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:01,175 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:01,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:01,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:01,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:01,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:01,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:01,175 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:01,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:01,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:01,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:01,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:01,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:01,176 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] throttle.PressureAwareThroughputController(145): 0337be0f939be2a2b593eefecee32758#C#compaction#541 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:56:01,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:01,177 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:01,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:01,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:01,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:01,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:01,180 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:01,180 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:01,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:01,180 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:01,180 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:01,180 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:01,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:01,180 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:56:01,180 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:56:01,180 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:01,181 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:01,181 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:01,181 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:01,181 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:01,181 DEBUG [Thread-3261] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:56:01,181 DEBUG [Thread-3261] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:56:01,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:01,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:01,181 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:56:01,197 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/C/1136fa759b5e46a38ba3804b086bbe5b as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/1136fa759b5e46a38ba3804b086bbe5b 2023-03-29 03:56:01,202 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 0337be0f939be2a2b593eefecee32758/C of 0337be0f939be2a2b593eefecee32758 into 1136fa759b5e46a38ba3804b086bbe5b(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:56:01,202 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0337be0f939be2a2b593eefecee32758: 2023-03-29 03:56:01,202 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758., storeName=0337be0f939be2a2b593eefecee32758/C, priority=11, startTime=1680062161101; duration=0sec 2023-03-29 03:56:01,203 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:56:01,252 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:01,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 288 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42004 deadline: 1680062221243, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:01,253 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:01,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42024 deadline: 1680062221243, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:01,253 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:01,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41942 deadline: 1680062221243, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:01,253 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:01,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 281 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42010 deadline: 1680062221245, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:01,281 DEBUG [Thread-3261] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:56:01,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:56:01,283 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:56:01,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:56:01,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:56:01,284 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:56:01,284 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:01,284 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:56:01,284 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:56:01,284 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:01,284 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:01,286 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:01,286 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:01,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:01,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:01,286 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:01,286 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:56:01,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:01,286 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:01,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:56:01,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:01,287 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:56:01,288 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:56:01,288 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:56:01,288 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:56:01,288 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:01,288 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:56:01,288 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:56:01,288 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:56:01,288 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:56:01,288 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. started... 2023-03-29 03:56:01,288 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. as already flushing 2023-03-29 03:56:01,288 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:56:01,288 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:56:01,288 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:56:01,289 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:01,289 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:56:01,289 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:56:01,290 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:01,290 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:01,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:01,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:01,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:01,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:01,291 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:01,291 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:56:01,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:01,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:01,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:01,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:01,294 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:56:01,294 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a6b7c50[Count = 0] remaining members to acquire global barrier 2023-03-29 03:56:01,294 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:56:01,294 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:01,298 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:01,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:01,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:01,299 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:56:01,299 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:56:01,299 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:56:01,300 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:01,300 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:56:01,301 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:01,301 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:56:01,301 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:01,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:01,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:01,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:01,301 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:56:01,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:01,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:01,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:01,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:01,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:01,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:01,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:01,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:56:01,304 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:56:01,304 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:56:01,304 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:56:01,304 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:56:01,304 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:56:01,304 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:56:01,306 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:01,306 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:01,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:01,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:01,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:01,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:01,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:01,306 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:01,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:01,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:01,307 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:01,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:01,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:01,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:01,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:01,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:01,309 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:01,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:01,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:01,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:01,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:01,311 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:01,311 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:01,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:01,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:01,312 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:01,312 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:56:01,312 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:56:01,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:01,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:01,312 DEBUG [Thread-3261] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:56:01,312 DEBUG [Thread-3261] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:56:01,312 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:01,312 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:01,313 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:01,313 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:01,313 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:01,313 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:01,313 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:01,313 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:56:01,413 DEBUG [Thread-3261] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:56:01,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:56:01,416 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:56:01,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:56:01,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:56:01,417 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:56:01,417 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:01,417 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:56:01,417 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:56:01,417 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:01,417 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:01,419 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:01,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:01,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:01,419 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:01,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:01,420 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:01,420 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:56:01,420 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:01,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:56:01,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:01,421 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:56:01,432 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:56:01,432 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:56:01,432 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:56:01,433 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:01,433 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:56:01,435 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:56:01,435 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:56:01,435 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:56:01,435 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. started... 2023-03-29 03:56:01,435 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. as already flushing 2023-03-29 03:56:01,435 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:56:01,435 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:56:01,435 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:56:01,435 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:01,435 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:56:01,435 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:56:01,437 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:01,437 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:01,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:01,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:01,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:01,437 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:01,437 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:56:01,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:01,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:01,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:01,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:01,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:01,439 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:56:01,439 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3f73019[Count = 0] remaining members to acquire global barrier 2023-03-29 03:56:01,439 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:56:01,439 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:01,440 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:01,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:01,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:01,441 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:56:01,441 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:56:01,441 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:56:01,441 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:01,441 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:56:01,443 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:01,443 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:56:01,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:01,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:01,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:01,443 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:01,444 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:56:01,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:01,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:01,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:01,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:01,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:01,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:01,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:01,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:56:01,447 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:56:01,447 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:56:01,447 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:56:01,447 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:56:01,447 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:56:01,447 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:56:01,449 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:01,449 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:01,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:01,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:01,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:01,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:01,449 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:01,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:01,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:01,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:01,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:01,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:01,450 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:01,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:01,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:01,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:01,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:01,451 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:01,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:01,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:01,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:01,453 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:01,453 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:01,453 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:01,453 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:01,453 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:01,453 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:56:01,454 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:56:01,454 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:01,454 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:01,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:01,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:01,454 DEBUG [Thread-3261] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:56:01,454 DEBUG [Thread-3261] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:56:01,453 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:01,453 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:01,454 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:01,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:01,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:01,459 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:01,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41942 deadline: 1680062221454, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:01,459 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:01,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 290 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42004 deadline: 1680062221455, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:01,460 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:01,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42024 deadline: 1680062221455, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:01,460 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:01,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 283 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42010 deadline: 1680062221455, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:01,540 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=544 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/A/174d05012b6c42c09dea0b6d8aff6366 2023-03-29 03:56:01,554 DEBUG [Thread-3261] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:56:01,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:56:01,556 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:56:01,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:56:01,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:56:01,557 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:56:01,557 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:01,557 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:56:01,557 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:56:01,558 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:01,558 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:01,561 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:01,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:01,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:01,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:01,561 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:01,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:56:01,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:01,562 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:56:01,564 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:01,564 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:01,564 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:56:01,580 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:56:01,580 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:56:01,580 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:56:01,580 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:01,580 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:56:01,582 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/A/7f05f3f8671042b49bca3bed9e75b114 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/7f05f3f8671042b49bca3bed9e75b114 2023-03-29 03:56:01,587 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 0337be0f939be2a2b593eefecee32758/A of 0337be0f939be2a2b593eefecee32758 into 7f05f3f8671042b49bca3bed9e75b114(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:56:01,587 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0337be0f939be2a2b593eefecee32758: 2023-03-29 03:56:01,587 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758., storeName=0337be0f939be2a2b593eefecee32758/A, priority=11, startTime=1680062161100; duration=0sec 2023-03-29 03:56:01,587 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:56:01,588 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:56:01,588 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:56:01,588 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:56:01,588 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. started... 2023-03-29 03:56:01,588 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. as already flushing 2023-03-29 03:56:01,588 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:56:01,588 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:56:01,588 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:56:01,588 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:01,591 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:56:01,591 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:56:01,594 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:01,594 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:01,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:01,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:01,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:01,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:01,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:01,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:01,596 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:01,596 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:56:01,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:01,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:01,597 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:56:01,597 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7eb0ec8e[Count = 0] remaining members to acquire global barrier 2023-03-29 03:56:01,597 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:56:01,597 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:01,598 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:01,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:01,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:01,599 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:01,599 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:56:01,599 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:56:01,599 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:56:01,599 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:56:01,601 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:01,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:01,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:01,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:01,601 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:56:01,601 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:01,601 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:56:01,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:01,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:01,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:01,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:01,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:01,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:01,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:01,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:56:01,605 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:56:01,605 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:56:01,605 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:56:01,605 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:56:01,605 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:56:01,605 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:56:01,608 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:01,608 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:01,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:01,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:01,608 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:01,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:01,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:01,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:01,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:01,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:01,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:01,609 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:01,610 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:01,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:01,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:01,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:01,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:01,613 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:01,613 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:01,613 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:01,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:01,613 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:01,613 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:01,613 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:01,613 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:01,613 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:56:01,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:01,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:01,613 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:56:01,613 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:01,614 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:01,613 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:01,614 DEBUG [Thread-3261] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:56:01,614 DEBUG [Thread-3261] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:56:01,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:01,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:01,614 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:56:01,714 DEBUG [Thread-3261] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:56:01,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:56:01,716 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:56:01,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:56:01,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:56:01,717 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:56:01,717 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:01,717 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:56:01,717 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:56:01,718 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:01,718 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:01,720 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:01,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:01,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:01,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:01,721 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:01,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:56:01,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:01,721 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:56:01,724 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:56:01,724 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:56:01,724 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:56:01,724 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:01,724 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:01,724 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:56:01,724 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:56:01,724 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:56:01,724 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:56:01,724 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:01,725 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:56:01,724 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. started... 2023-03-29 03:56:01,725 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. as already flushing 2023-03-29 03:56:01,725 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:56:01,725 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:56:01,725 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:56:01,725 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:01,725 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:56:01,725 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:56:01,726 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:01,726 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:01,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:01,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:01,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:01,727 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:01,727 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:56:01,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:01,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:01,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:01,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:01,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:01,730 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:56:01,730 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@68d2c0b9[Count = 0] remaining members to acquire global barrier 2023-03-29 03:56:01,730 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:56:01,730 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:01,731 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:01,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:01,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:01,731 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:56:01,732 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:56:01,732 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:56:01,732 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:01,732 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:56:01,738 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:56:01,739 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:01,739 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:56:01,739 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:01,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:01,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:01,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:01,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:01,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:01,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:01,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:01,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:01,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:01,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:01,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:56:01,742 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:56:01,742 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:56:01,742 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:56:01,742 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:56:01,742 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:56:01,742 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:56:01,744 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:01,744 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:01,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:01,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:01,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:01,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:01,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:01,744 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:01,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:01,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:01,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:01,745 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:01,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:01,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:01,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:01,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:01,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:01,758 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:01,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:01,766 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:01,766 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:01,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 292 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42004 deadline: 1680062221761, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:01,766 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:01,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:01,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:01,766 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:01,766 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:01,766 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:01,767 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:01,767 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:01,766 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:01,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41942 deadline: 1680062221761, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:01,767 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:56:01,767 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:56:01,767 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:01,767 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:01,767 DEBUG [Thread-3261] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:56:01,767 DEBUG [Thread-3261] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:56:01,767 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:01,768 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:01,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42024 deadline: 1680062221763, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:01,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:01,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:01,768 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:56:01,768 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:01,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 285 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42010 deadline: 1680062221763, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:01,868 DEBUG [Thread-3261] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:56:01,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:56:01,870 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:56:01,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:56:01,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:56:01,871 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:56:01,871 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:01,871 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:56:01,871 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:56:01,872 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:01,872 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:01,874 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:01,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:01,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:01,874 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:01,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:01,875 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:01,875 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:56:01,875 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:01,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:56:01,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:01,875 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:56:01,876 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:56:01,876 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:56:01,876 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:01,876 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:56:01,877 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:56:01,877 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:56:01,877 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:56:01,877 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. started... 2023-03-29 03:56:01,877 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. as already flushing 2023-03-29 03:56:01,877 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:56:01,877 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:56:01,877 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:56:01,877 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:01,877 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:56:01,877 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:56:01,879 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:01,879 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:01,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:01,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:01,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:01,879 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:01,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:01,879 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:56:01,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:01,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:01,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:01,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:01,882 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:56:01,882 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@695aa80f[Count = 0] remaining members to acquire global barrier 2023-03-29 03:56:01,882 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:56:01,882 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:01,883 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:01,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:01,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:01,884 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:56:01,884 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:56:01,884 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:56:01,885 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:01,885 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:56:01,886 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:01,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:01,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:01,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:01,886 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:56:01,886 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:01,887 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:56:01,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:01,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:01,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:01,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:01,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:01,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:01,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:01,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:56:01,890 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:56:01,890 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:56:01,891 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:56:01,891 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:56:01,891 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:56:01,891 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:56:01,894 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:01,894 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:01,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:01,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:01,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:01,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:01,894 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:01,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:01,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:01,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:01,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:01,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:01,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:01,895 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:01,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:01,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:01,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:01,899 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:01,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:01,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:01,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:01,901 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:01,901 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:01,902 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:01,902 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:01,902 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:01,902 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:01,902 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:01,902 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:01,902 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:01,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:01,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:01,902 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:56:01,902 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:56:01,902 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:01,902 DEBUG [Thread-3261] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:56:01,902 DEBUG [Thread-3261] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:56:01,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:01,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:01,994 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=544 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/B/4c89102314124fe6873b294cbb23270e 2023-03-29 03:56:02,002 DEBUG [Thread-3261] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:56:02,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:56:02,005 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:56:02,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:56:02,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:56:02,008 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:56:02,008 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:02,008 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:56:02,008 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:56:02,008 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:02,008 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:02,010 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:02,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:02,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:02,010 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:02,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:02,012 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:02,012 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:02,012 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:56:02,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:56:02,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:02,012 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:56:02,013 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:56:02,013 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:56:02,013 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:56:02,013 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:02,013 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:56:02,013 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:56:02,013 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:56:02,013 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:56:02,013 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. started... 2023-03-29 03:56:02,013 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. as already flushing 2023-03-29 03:56:02,013 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:56:02,013 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:56:02,013 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:56:02,014 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:02,014 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:56:02,014 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:56:02,015 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:02,015 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:02,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:02,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:02,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:02,016 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:02,016 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:56:02,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:02,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:02,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:02,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:02,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:02,019 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:56:02,019 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@53173ac5[Count = 0] remaining members to acquire global barrier 2023-03-29 03:56:02,019 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:56:02,019 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:02,020 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:02,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:02,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:02,021 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:56:02,021 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:56:02,021 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:02,021 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:56:02,021 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:56:02,022 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=544 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/C/1b0a9b5703d44107868249c7c14ea58a 2023-03-29 03:56:02,023 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:56:02,024 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:02,024 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:56:02,024 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:02,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:02,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:02,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:02,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:02,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:02,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:02,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:02,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:02,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:02,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:02,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:56:02,030 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:56:02,030 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:56:02,030 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:56:02,030 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:56:02,030 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:56:02,030 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:56:02,030 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/A/174d05012b6c42c09dea0b6d8aff6366 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/174d05012b6c42c09dea0b6d8aff6366 2023-03-29 03:56:02,032 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:02,032 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:02,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:02,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:02,032 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:02,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:02,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:02,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:02,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:02,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:02,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:02,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:02,035 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:02,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:02,035 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/174d05012b6c42c09dea0b6d8aff6366, entries=200, sequenceid=544, filesize=14.3 K 2023-03-29 03:56:02,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:02,036 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/B/4c89102314124fe6873b294cbb23270e as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/4c89102314124fe6873b294cbb23270e 2023-03-29 03:56:02,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:02,037 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:02,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:02,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:02,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:02,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:02,039 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/4c89102314124fe6873b294cbb23270e, entries=150, sequenceid=544, filesize=11.9 K 2023-03-29 03:56:02,040 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/C/1b0a9b5703d44107868249c7c14ea58a as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/1b0a9b5703d44107868249c7c14ea58a 2023-03-29 03:56:02,041 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:02,041 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:02,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:02,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:02,041 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:56:02,041 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:02,041 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:02,041 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:02,041 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:02,041 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:02,041 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:02,041 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:02,045 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/1b0a9b5703d44107868249c7c14ea58a, entries=150, sequenceid=544, filesize=11.9 K 2023-03-29 03:56:02,045 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~405.56 KB/415296, currentSize=46.96 KB/48090 for 0337be0f939be2a2b593eefecee32758 in 935ms, sequenceid=544, compaction requested=false 2023-03-29 03:56:02,045 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 0337be0f939be2a2b593eefecee32758: 2023-03-29 03:56:02,041 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:56:02,048 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:56:02,041 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:02,048 DEBUG [Thread-3261] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:56:02,048 DEBUG [Thread-3261] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:56:02,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:02,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:02,148 DEBUG [Thread-3261] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:56:02,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:56:02,150 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:56:02,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:56:02,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:56:02,152 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:56:02,152 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:02,152 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:56:02,152 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:56:02,152 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:02,152 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:02,154 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:02,154 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:02,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:02,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:02,155 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:02,155 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:56:02,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:02,155 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:02,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:56:02,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:02,156 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:56:02,156 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:56:02,156 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:56:02,156 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:56:02,156 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:02,156 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:56:02,156 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:56:02,156 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:56:02,156 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:56:02,156 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. started... 2023-03-29 03:56:02,157 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0337be0f939be2a2b593eefecee32758 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-29 03:56:02,157 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0337be0f939be2a2b593eefecee32758, store=A 2023-03-29 03:56:02,157 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:56:02,157 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0337be0f939be2a2b593eefecee32758, store=B 2023-03-29 03:56:02,157 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:56:02,157 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0337be0f939be2a2b593eefecee32758, store=C 2023-03-29 03:56:02,157 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:56:02,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on 0337be0f939be2a2b593eefecee32758 2023-03-29 03:56:02,271 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. as already flushing 2023-03-29 03:56:02,298 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680062162284, max timestamp=1680062162296Num uniques 859; 2023-03-29 03:56:02,298 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680062162284, max timestamp=1680062162296Num uniques 859; 2023-03-29 03:56:02,298 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680062162284, max timestamp=1680062162296Num uniques 859; 2023-03-29 03:56:02,298 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-29 03:56:02,298 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-29 03:56:02,298 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-29 03:56:02,330 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:02,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 302 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42004 deadline: 1680062222323, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:02,331 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:02,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42024 deadline: 1680062222324, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:02,332 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:02,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41942 deadline: 1680062222329, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:02,338 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:02,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 295 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42010 deadline: 1680062222330, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:02,435 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:02,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 304 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42004 deadline: 1680062222431, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:02,435 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:02,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42024 deadline: 1680062222432, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:02,436 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:02,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41942 deadline: 1680062222433, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:02,447 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:02,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 297 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42010 deadline: 1680062222439, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:02,575 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=557 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/A/c345e9e534f442c094c75bbd109fbeaa 2023-03-29 03:56:02,591 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=557 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/B/623594276a2e45c2be7f2fde45197a17 2023-03-29 03:56:02,607 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=557 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/C/e6bdd06d64a7451d9f26308f0e362f63 2023-03-29 03:56:02,612 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/A/c345e9e534f442c094c75bbd109fbeaa as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/c345e9e534f442c094c75bbd109fbeaa 2023-03-29 03:56:02,617 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/c345e9e534f442c094c75bbd109fbeaa, entries=150, sequenceid=557, filesize=11.9 K 2023-03-29 03:56:02,617 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/B/623594276a2e45c2be7f2fde45197a17 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/623594276a2e45c2be7f2fde45197a17 2023-03-29 03:56:02,620 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/623594276a2e45c2be7f2fde45197a17, entries=150, sequenceid=557, filesize=11.9 K 2023-03-29 03:56:02,621 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/C/e6bdd06d64a7451d9f26308f0e362f63 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/e6bdd06d64a7451d9f26308f0e362f63 2023-03-29 03:56:02,625 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/e6bdd06d64a7451d9f26308f0e362f63, entries=150, sequenceid=557, filesize=11.9 K 2023-03-29 03:56:02,626 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 0337be0f939be2a2b593eefecee32758 in 469ms, sequenceid=557, compaction requested=true 2023-03-29 03:56:02,626 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0337be0f939be2a2b593eefecee32758: 2023-03-29 03:56:02,626 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:56:02,626 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:56:02,627 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:56:02,627 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:02,627 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:56:02,627 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:56:02,629 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:02,630 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:02,630 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:56:02,636 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:02,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:02,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:02,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:02,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:02,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:02,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:02,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:02,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:02,639 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:56:02,639 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2aba68b5[Count = 0] remaining members to acquire global barrier 2023-03-29 03:56:02,639 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:56:02,639 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:02,640 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:02,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:02,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:02,640 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:02,640 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:56:02,640 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:56:02,640 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:56:02,640 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:56:02,641 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:56:02,642 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:02,642 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:56:02,642 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:02,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:02,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:02,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:02,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:02,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:02,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:02,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:02,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:02,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:02,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:02,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:56:02,645 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:56:02,645 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:56:02,645 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:56:02,645 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:56:02,645 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:56:02,645 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:56:02,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on 0337be0f939be2a2b593eefecee32758 2023-03-29 03:56:02,646 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 0337be0f939be2a2b593eefecee32758 3/3 column families, dataSize=140.89 KB heapSize=395.53 KB 2023-03-29 03:56:02,647 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0337be0f939be2a2b593eefecee32758, store=A 2023-03-29 03:56:02,647 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-29 03:56:02,647 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0337be0f939be2a2b593eefecee32758, store=B 2023-03-29 03:56:02,647 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-29 03:56:02,647 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0337be0f939be2a2b593eefecee32758, store=C 2023-03-29 03:56:02,647 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-29 03:56:02,648 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:02,648 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:02,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:02,648 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:02,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:02,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:02,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:02,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:02,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:02,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:02,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:02,649 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:02,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:02,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:02,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:02,653 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:02,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:02,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:02,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:02,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:02,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:02,658 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:02,658 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:02,658 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:02,658 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:02,658 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:02,658 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:02,658 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:02,658 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:02,658 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:56:02,658 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:56:02,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:02,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:02,658 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:02,659 DEBUG [Thread-3261] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:56:02,659 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:56:02,658 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:02,659 DEBUG [Thread-3261] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:56:02,659 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:02,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:02,666 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:02,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 308 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42004 deadline: 1680062222657, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:02,667 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:02,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41942 deadline: 1680062222658, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:02,667 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:02,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 300 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42010 deadline: 1680062222659, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:02,668 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=594 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/A/8ddf6e37a8d7407cafe2ff2b319c2c93 2023-03-29 03:56:02,672 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:02,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42024 deadline: 1680062222660, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:02,682 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=594 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/B/63face3b15d34951ab280bce5d0248c8 2023-03-29 03:56:02,693 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=594 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/C/bab00af85694494bbf8b1c24a2e43df7 2023-03-29 03:56:02,697 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/A/8ddf6e37a8d7407cafe2ff2b319c2c93 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/8ddf6e37a8d7407cafe2ff2b319c2c93 2023-03-29 03:56:02,702 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/8ddf6e37a8d7407cafe2ff2b319c2c93, entries=250, sequenceid=594, filesize=16.7 K 2023-03-29 03:56:02,702 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/B/63face3b15d34951ab280bce5d0248c8 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/63face3b15d34951ab280bce5d0248c8 2023-03-29 03:56:02,705 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/63face3b15d34951ab280bce5d0248c8, entries=150, sequenceid=594, filesize=11.9 K 2023-03-29 03:56:02,707 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/C/bab00af85694494bbf8b1c24a2e43df7 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/bab00af85694494bbf8b1c24a2e43df7 2023-03-29 03:56:02,713 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/bab00af85694494bbf8b1c24a2e43df7, entries=150, sequenceid=594, filesize=11.9 K 2023-03-29 03:56:02,714 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~415.41 KB/425376, currentSize=33.54 KB/34350 for 0337be0f939be2a2b593eefecee32758 in 67ms, sequenceid=594, compaction requested=true 2023-03-29 03:56:02,714 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 0337be0f939be2a2b593eefecee32758: 2023-03-29 03:56:02,714 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:56:02,714 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:56:02,714 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-29 03:56:02,714 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-29 03:56:02,714 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-29 03:56:02,715 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 57324 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-29 03:56:02,715 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1912): 0337be0f939be2a2b593eefecee32758/A is initiating minor compaction (all files) 2023-03-29 03:56:02,715 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0337be0f939be2a2b593eefecee32758/A in TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:56:02,715 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/7f05f3f8671042b49bca3bed9e75b114, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/174d05012b6c42c09dea0b6d8aff6366, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/c345e9e534f442c094c75bbd109fbeaa, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/8ddf6e37a8d7407cafe2ff2b319c2c93] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp, totalSize=56.0 K 2023-03-29 03:56:02,716 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 50004 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-29 03:56:02,716 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1912): 0337be0f939be2a2b593eefecee32758/B is initiating minor compaction (all files) 2023-03-29 03:56:02,716 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 0337be0f939be2a2b593eefecee32758/B in TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:56:02,716 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/668d41bbbf754f38883ac62f93892179, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/4c89102314124fe6873b294cbb23270e, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/623594276a2e45c2be7f2fde45197a17, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/63face3b15d34951ab280bce5d0248c8] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp, totalSize=48.8 K 2023-03-29 03:56:02,716 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 7f05f3f8671042b49bca3bed9e75b114, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=506, earliestPutTs=1680062160622 2023-03-29 03:56:02,716 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 668d41bbbf754f38883ac62f93892179, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=506, earliestPutTs=1680062160622 2023-03-29 03:56:02,717 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 4c89102314124fe6873b294cbb23270e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=544, earliestPutTs=1680062160753 2023-03-29 03:56:02,717 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 174d05012b6c42c09dea0b6d8aff6366, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=544, earliestPutTs=1680062160753 2023-03-29 03:56:02,717 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 623594276a2e45c2be7f2fde45197a17, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=557, earliestPutTs=1680062161130 2023-03-29 03:56:02,717 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting c345e9e534f442c094c75bbd109fbeaa, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=557, earliestPutTs=1680062161130 2023-03-29 03:56:02,718 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 63face3b15d34951ab280bce5d0248c8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=594, earliestPutTs=1680062162322 2023-03-29 03:56:02,718 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 8ddf6e37a8d7407cafe2ff2b319c2c93, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=594, earliestPutTs=1680062162322 2023-03-29 03:56:02,729 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0337be0f939be2a2b593eefecee32758#B#compaction#550 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:56:02,738 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] throttle.PressureAwareThroughputController(145): 0337be0f939be2a2b593eefecee32758#A#compaction#551 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:56:02,759 DEBUG [Thread-3261] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:56:02,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:56:02,761 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:56:02,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on 0337be0f939be2a2b593eefecee32758 2023-03-29 03:56:02,773 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 0337be0f939be2a2b593eefecee32758 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-29 03:56:02,773 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0337be0f939be2a2b593eefecee32758, store=A 2023-03-29 03:56:02,774 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:56:02,774 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0337be0f939be2a2b593eefecee32758, store=B 2023-03-29 03:56:02,774 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:56:02,774 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0337be0f939be2a2b593eefecee32758, store=C 2023-03-29 03:56:02,774 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:56:02,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:56:02,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:56:02,776 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:56:02,776 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:02,776 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:56:02,776 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:56:02,777 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:02,777 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:02,778 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/B/989d7e0a0235425e80c6cf199cf2fd91 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/989d7e0a0235425e80c6cf199cf2fd91 2023-03-29 03:56:02,778 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:02,778 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:02,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:02,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:02,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:02,779 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:02,779 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:56:02,779 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:02,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:56:02,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:02,779 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:56:02,782 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 0337be0f939be2a2b593eefecee32758/B of 0337be0f939be2a2b593eefecee32758 into 989d7e0a0235425e80c6cf199cf2fd91(size=13.3 K), total size for store is 13.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:56:02,782 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0337be0f939be2a2b593eefecee32758: 2023-03-29 03:56:02,782 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758., storeName=0337be0f939be2a2b593eefecee32758/B, priority=12, startTime=1680062162714; duration=0sec 2023-03-29 03:56:02,782 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:56:02,782 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-29 03:56:02,783 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 50004 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-29 03:56:02,783 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1912): 0337be0f939be2a2b593eefecee32758/C is initiating minor compaction (all files) 2023-03-29 03:56:02,783 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 0337be0f939be2a2b593eefecee32758/C in TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:56:02,783 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/1136fa759b5e46a38ba3804b086bbe5b, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/1b0a9b5703d44107868249c7c14ea58a, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/e6bdd06d64a7451d9f26308f0e362f63, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/bab00af85694494bbf8b1c24a2e43df7] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp, totalSize=48.8 K 2023-03-29 03:56:02,784 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 1136fa759b5e46a38ba3804b086bbe5b, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=506, earliestPutTs=1680062160622 2023-03-29 03:56:02,784 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 1b0a9b5703d44107868249c7c14ea58a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=544, earliestPutTs=1680062160753 2023-03-29 03:56:02,785 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting e6bdd06d64a7451d9f26308f0e362f63, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=557, earliestPutTs=1680062161130 2023-03-29 03:56:02,786 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting bab00af85694494bbf8b1c24a2e43df7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=594, earliestPutTs=1680062162322 2023-03-29 03:56:02,788 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:56:02,788 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:56:02,788 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:56:02,788 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:02,788 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:56:02,788 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:56:02,788 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:56:02,788 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:56:02,789 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. started... 2023-03-29 03:56:02,789 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. as already flushing 2023-03-29 03:56:02,789 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:56:02,789 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:56:02,789 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:56:02,789 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:02,789 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:56:02,789 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:56:02,791 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:02,791 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:02,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:02,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:02,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:02,791 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:02,791 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:56:02,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:02,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:02,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:02,792 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0337be0f939be2a2b593eefecee32758#C#compaction#553 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:56:02,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:02,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:02,793 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/A/cfbc79711a644e11975b0b79e7bfc352 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/cfbc79711a644e11975b0b79e7bfc352 2023-03-29 03:56:02,793 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:56:02,793 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5413a253[Count = 0] remaining members to acquire global barrier 2023-03-29 03:56:02,793 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:56:02,793 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:02,796 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:02,796 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:02,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:02,796 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:56:02,796 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:56:02,796 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:56:02,796 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:02,796 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:56:02,797 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:56:02,797 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:02,797 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:56:02,798 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:02,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:02,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:02,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:02,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:02,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:02,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:02,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:02,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:02,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:02,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:02,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:56:02,801 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:56:02,801 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:56:02,801 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:56:02,801 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:56:02,801 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:56:02,801 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:56:02,801 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 0337be0f939be2a2b593eefecee32758/A of 0337be0f939be2a2b593eefecee32758 into cfbc79711a644e11975b0b79e7bfc352(size=13.3 K), total size for store is 13.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:56:02,801 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0337be0f939be2a2b593eefecee32758: 2023-03-29 03:56:02,801 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758., storeName=0337be0f939be2a2b593eefecee32758/A, priority=12, startTime=1680062162714; duration=0sec 2023-03-29 03:56:02,802 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:56:02,803 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:02,803 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:02,803 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:02,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:02,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:02,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:02,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:02,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:02,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:02,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:02,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:02,804 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:02,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:02,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:02,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:02,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:02,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:02,805 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:02,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:02,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:02,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:02,807 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:02,807 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:02,807 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:56:02,807 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:56:02,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:02,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:02,807 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:02,807 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:02,807 DEBUG [Thread-3261] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:56:02,807 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:02,807 DEBUG [Thread-3261] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:56:02,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:02,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:02,807 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:02,807 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:02,807 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:02,807 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:02,807 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:02,808 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=604 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/A/d34ed3d3a6ca43ca95085bfd5f40febf 2023-03-29 03:56:02,823 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=604 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/B/e0c058ab07614f989f11027815697f91 2023-03-29 03:56:02,836 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=604 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/C/3db623b7a43d4f1c85f6275ca14b09b7 2023-03-29 03:56:02,839 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680062162813, max timestamp=1680062162837Num uniques 859; 2023-03-29 03:56:02,839 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680062162813, max timestamp=1680062162837Num uniques 859; 2023-03-29 03:56:02,839 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-29 03:56:02,839 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-29 03:56:02,839 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680062162813, max timestamp=1680062162837Num uniques 859; 2023-03-29 03:56:02,840 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-29 03:56:02,846 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/A/d34ed3d3a6ca43ca95085bfd5f40febf as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/d34ed3d3a6ca43ca95085bfd5f40febf 2023-03-29 03:56:02,849 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/d34ed3d3a6ca43ca95085bfd5f40febf, entries=200, sequenceid=604, filesize=14.3 K 2023-03-29 03:56:02,850 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/B/e0c058ab07614f989f11027815697f91 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/e0c058ab07614f989f11027815697f91 2023-03-29 03:56:02,853 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/e0c058ab07614f989f11027815697f91, entries=100, sequenceid=604, filesize=9.5 K 2023-03-29 03:56:02,853 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/C/3db623b7a43d4f1c85f6275ca14b09b7 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/3db623b7a43d4f1c85f6275ca14b09b7 2023-03-29 03:56:02,861 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/3db623b7a43d4f1c85f6275ca14b09b7, entries=100, sequenceid=604, filesize=9.5 K 2023-03-29 03:56:02,872 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=46.96 KB/48090 for 0337be0f939be2a2b593eefecee32758 in 99ms, sequenceid=604, compaction requested=false 2023-03-29 03:56:02,872 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 0337be0f939be2a2b593eefecee32758: 2023-03-29 03:56:02,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on 0337be0f939be2a2b593eefecee32758 2023-03-29 03:56:02,872 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 0337be0f939be2a2b593eefecee32758 3/3 column families, dataSize=67.09 KB heapSize=176.39 KB 2023-03-29 03:56:02,872 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0337be0f939be2a2b593eefecee32758, store=A 2023-03-29 03:56:02,872 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-29 03:56:02,873 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0337be0f939be2a2b593eefecee32758, store=B 2023-03-29 03:56:02,873 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-29 03:56:02,873 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0337be0f939be2a2b593eefecee32758, store=C 2023-03-29 03:56:02,873 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-29 03:56:02,885 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=631 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/A/a8a62a6bd82245178e88df4cd7b26ad1 2023-03-29 03:56:02,902 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=631 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/B/e2ad3864577a4fd3ae618133e5e9adc1 2023-03-29 03:56:02,907 DEBUG [Thread-3261] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:56:02,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:56:02,909 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:56:02,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:56:02,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:56:02,916 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:56:02,916 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:02,916 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:56:02,916 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:56:02,916 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:02,916 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:02,921 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:02,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:02,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:02,921 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:02,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:02,921 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:02,921 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:56:02,921 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:02,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:56:02,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:02,922 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:56:02,923 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:56:02,923 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:56:02,923 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:56:02,923 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:02,923 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:56:02,924 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:02,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 321 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42004 deadline: 1680062222912, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:02,925 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:02,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41942 deadline: 1680062222920, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:02,928 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:56:02,928 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:56:02,928 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:56:02,928 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. started... 2023-03-29 03:56:02,928 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=631 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/C/564227d153394044a02da50f51902990 2023-03-29 03:56:02,928 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. as already flushing 2023-03-29 03:56:02,929 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:56:02,929 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:56:02,929 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:56:02,929 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:02,929 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:56:02,929 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:56:02,929 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:02,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 312 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42010 deadline: 1680062222921, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:02,931 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:02,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:02,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:02,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:02,931 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:02,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:02,931 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:02,931 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:56:02,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:02,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:02,934 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:02,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42024 deadline: 1680062222924, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:02,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:02,935 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/A/a8a62a6bd82245178e88df4cd7b26ad1 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/a8a62a6bd82245178e88df4cd7b26ad1 2023-03-29 03:56:02,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:02,935 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:56:02,935 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6835720e[Count = 0] remaining members to acquire global barrier 2023-03-29 03:56:02,935 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:56:02,935 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:02,937 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:02,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:02,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:02,937 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:02,937 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:56:02,937 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:56:02,937 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:56:02,937 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:56:02,939 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:56:02,939 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:02,939 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:56:02,939 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:02,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:02,939 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/a8a62a6bd82245178e88df4cd7b26ad1, entries=150, sequenceid=631, filesize=11.9 K 2023-03-29 03:56:02,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:02,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:02,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:02,940 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/B/e2ad3864577a4fd3ae618133e5e9adc1 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/e2ad3864577a4fd3ae618133e5e9adc1 2023-03-29 03:56:02,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:02,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:02,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:02,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:02,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:02,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:02,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:56:02,942 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:56:02,942 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:56:02,942 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:56:02,942 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:56:02,942 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:56:02,942 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:56:02,943 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/e2ad3864577a4fd3ae618133e5e9adc1, entries=150, sequenceid=631, filesize=11.9 K 2023-03-29 03:56:02,943 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/C/564227d153394044a02da50f51902990 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/564227d153394044a02da50f51902990 2023-03-29 03:56:02,946 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:02,946 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:02,946 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:02,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:02,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:02,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:02,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:02,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:02,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:02,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:02,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:02,946 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:02,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:02,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:02,947 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/564227d153394044a02da50f51902990, entries=150, sequenceid=631, filesize=11.9 K 2023-03-29 03:56:02,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:02,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:02,948 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:02,948 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~176.34 KB/180576, currentSize=114.05 KB/116790 for 0337be0f939be2a2b593eefecee32758 in 76ms, sequenceid=631, compaction requested=true 2023-03-29 03:56:02,948 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 0337be0f939be2a2b593eefecee32758: 2023-03-29 03:56:02,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:02,948 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:56:02,948 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:56:02,948 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-29 03:56:02,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:02,948 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-29 03:56:02,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:02,949 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 40390 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-29 03:56:02,949 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1912): 0337be0f939be2a2b593eefecee32758/A is initiating minor compaction (all files) 2023-03-29 03:56:02,949 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0337be0f939be2a2b593eefecee32758/A in TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:56:02,949 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/cfbc79711a644e11975b0b79e7bfc352, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/d34ed3d3a6ca43ca95085bfd5f40febf, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/a8a62a6bd82245178e88df4cd7b26ad1] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp, totalSize=39.4 K 2023-03-29 03:56:02,950 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting cfbc79711a644e11975b0b79e7bfc352, keycount=150, bloomtype=ROW, size=13.3 K, encoding=NONE, compression=NONE, seqNum=594, earliestPutTs=1680062162322 2023-03-29 03:56:02,950 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting d34ed3d3a6ca43ca95085bfd5f40febf, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=604, earliestPutTs=1680062162657 2023-03-29 03:56:02,950 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:02,951 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:02,951 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:02,951 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:02,951 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:56:02,951 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:56:02,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:02,951 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:02,951 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:02,951 DEBUG [Thread-3261] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:56:02,951 DEBUG [Thread-3261] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:56:02,951 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:02,951 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:02,951 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting a8a62a6bd82245178e88df4cd7b26ad1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=631, earliestPutTs=1680062162844 2023-03-29 03:56:02,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:02,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:02,951 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:02,951 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:02,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:02,951 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:56:02,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:02,956 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] throttle.PressureAwareThroughputController(145): 0337be0f939be2a2b593eefecee32758#A#compaction#559 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:56:02,967 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/A/5d72012d115b4fa094ec80ef59eed0eb as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/5d72012d115b4fa094ec80ef59eed0eb 2023-03-29 03:56:02,971 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0337be0f939be2a2b593eefecee32758/A of 0337be0f939be2a2b593eefecee32758 into 5d72012d115b4fa094ec80ef59eed0eb(size=13.4 K), total size for store is 13.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:56:02,971 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0337be0f939be2a2b593eefecee32758: 2023-03-29 03:56:02,971 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758., storeName=0337be0f939be2a2b593eefecee32758/A, priority=13, startTime=1680062162948; duration=0sec 2023-03-29 03:56:02,971 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-29 03:56:02,971 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 4 compacting, 2 eligible, 16 blocking 2023-03-29 03:56:02,972 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-03-29 03:56:02,972 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-03-29 03:56:02,972 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. because compaction request was cancelled 2023-03-29 03:56:02,972 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-29 03:56:02,974 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35510 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-29 03:56:02,974 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1912): 0337be0f939be2a2b593eefecee32758/B is initiating minor compaction (all files) 2023-03-29 03:56:02,974 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0337be0f939be2a2b593eefecee32758/B in TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:56:02,974 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/989d7e0a0235425e80c6cf199cf2fd91, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/e0c058ab07614f989f11027815697f91, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/e2ad3864577a4fd3ae618133e5e9adc1] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp, totalSize=34.7 K 2023-03-29 03:56:02,974 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 989d7e0a0235425e80c6cf199cf2fd91, keycount=150, bloomtype=ROW, size=13.3 K, encoding=NONE, compression=NONE, seqNum=594, earliestPutTs=1680062162322 2023-03-29 03:56:02,975 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting e0c058ab07614f989f11027815697f91, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=604, earliestPutTs=1680062162657 2023-03-29 03:56:02,975 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting e2ad3864577a4fd3ae618133e5e9adc1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=631, earliestPutTs=1680062162844 2023-03-29 03:56:02,982 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] throttle.PressureAwareThroughputController(145): 0337be0f939be2a2b593eefecee32758#B#compaction#560 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:56:02,998 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/B/13a6ca1c587a49a2b16109f0e4e8a872 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/13a6ca1c587a49a2b16109f0e4e8a872 2023-03-29 03:56:03,005 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0337be0f939be2a2b593eefecee32758/B of 0337be0f939be2a2b593eefecee32758 into 13a6ca1c587a49a2b16109f0e4e8a872(size=13.4 K), total size for store is 13.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:56:03,005 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0337be0f939be2a2b593eefecee32758: 2023-03-29 03:56:03,005 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758., storeName=0337be0f939be2a2b593eefecee32758/B, priority=13, startTime=1680062162948; duration=0sec 2023-03-29 03:56:03,005 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:56:03,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on 0337be0f939be2a2b593eefecee32758 2023-03-29 03:56:03,029 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 0337be0f939be2a2b593eefecee32758 3/3 column families, dataSize=120.76 KB heapSize=360.05 KB 2023-03-29 03:56:03,029 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0337be0f939be2a2b593eefecee32758, store=A 2023-03-29 03:56:03,029 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-29 03:56:03,029 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0337be0f939be2a2b593eefecee32758, store=B 2023-03-29 03:56:03,029 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-29 03:56:03,029 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0337be0f939be2a2b593eefecee32758, store=C 2023-03-29 03:56:03,029 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-29 03:56:03,050 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:03,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 316 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42010 deadline: 1680062223039, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:03,050 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:03,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41942 deadline: 1680062223040, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:03,051 DEBUG [Thread-3261] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:56:03,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:56:03,053 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:56:03,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:56:03,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:56:03,054 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:56:03,054 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:03,054 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:56:03,054 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:56:03,055 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:03,055 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:03,056 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:03,056 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:03,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:03,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:03,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:03,057 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:03,057 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:56:03,057 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:03,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:56:03,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:03,057 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:56:03,059 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:03,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 325 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42004 deadline: 1680062223049, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:03,059 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:03,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42024 deadline: 1680062223050, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:03,072 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:56:03,072 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:56:03,072 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:56:03,072 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:03,072 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:56:03,072 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:56:03,072 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:56:03,072 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:56:03,073 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. started... 2023-03-29 03:56:03,073 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. as already flushing 2023-03-29 03:56:03,073 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:56:03,073 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:56:03,073 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:56:03,073 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:03,073 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:56:03,073 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:56:03,075 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:03,075 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:03,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:03,075 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:03,075 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:56:03,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:03,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:03,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:03,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:03,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:03,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:03,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:03,077 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:56:03,077 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@29ebcdf[Count = 0] remaining members to acquire global barrier 2023-03-29 03:56:03,077 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:56:03,077 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:03,079 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:03,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:03,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:03,079 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:56:03,080 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:56:03,080 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:56:03,080 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:03,080 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:56:03,081 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:56:03,081 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:03,081 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:56:03,081 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:03,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:03,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:03,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:03,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:03,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:03,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:03,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:03,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:03,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:03,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:03,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:56:03,086 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:56:03,086 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:56:03,086 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:56:03,086 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:56:03,086 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:56:03,086 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:56:03,089 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:03,089 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:03,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:03,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:03,089 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:03,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:03,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:03,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:03,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:03,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:03,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:03,090 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:03,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:03,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:03,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:03,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:03,091 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:03,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:03,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:03,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:03,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:03,099 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:03,099 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:03,099 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:03,099 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:03,099 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:56:03,099 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:56:03,099 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:03,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:03,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:03,099 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:03,099 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:03,099 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:03,099 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:03,099 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:03,099 DEBUG [Thread-3261] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:56:03,099 DEBUG [Thread-3261] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:56:03,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:03,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:03,100 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:56:03,155 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:03,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 318 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42010 deadline: 1680062223151, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:03,156 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:03,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41942 deadline: 1680062223151, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:03,168 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:03,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42024 deadline: 1680062223161, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:03,169 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:03,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 327 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42004 deadline: 1680062223161, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:03,199 DEBUG [Thread-3261] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:56:03,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:56:03,202 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:56:03,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:56:03,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:56:03,203 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:56:03,203 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:03,203 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:56:03,203 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:56:03,203 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:03,204 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:03,205 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:03,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:03,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:03,205 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:03,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:03,205 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:03,205 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:56:03,206 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:03,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:56:03,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:03,206 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:56:03,206 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:56:03,206 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:56:03,206 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:56:03,206 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:03,207 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:56:03,208 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:56:03,208 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:56:03,208 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:56:03,208 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. started... 2023-03-29 03:56:03,208 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. as already flushing 2023-03-29 03:56:03,208 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:56:03,208 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:56:03,208 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:56:03,208 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:03,208 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:56:03,208 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:56:03,211 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:03,211 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:03,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:03,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:03,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:03,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:03,211 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:03,211 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:56:03,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:03,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:03,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:03,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:03,213 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:56:03,213 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b1893f9[Count = 0] remaining members to acquire global barrier 2023-03-29 03:56:03,213 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:56:03,213 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:03,217 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:03,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:03,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:03,217 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:56:03,217 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:56:03,217 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:56:03,219 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:03,219 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:56:03,220 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:56:03,220 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:03,220 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:03,220 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:56:03,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:03,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:03,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:03,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:03,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:03,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:03,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:03,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:03,223 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/C/109e3df5aa334d31b94d54b75a7f09cf as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/109e3df5aa334d31b94d54b75a7f09cf 2023-03-29 03:56:03,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:03,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:03,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:56:03,225 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:56:03,225 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:56:03,225 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:56:03,225 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:56:03,225 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:56:03,225 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:56:03,227 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:03,227 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:03,227 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:03,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:03,227 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:03,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:03,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:03,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:03,227 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:03,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:03,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:03,229 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 0337be0f939be2a2b593eefecee32758/C of 0337be0f939be2a2b593eefecee32758 into 109e3df5aa334d31b94d54b75a7f09cf(size=13.3 K), total size for store is 34.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:56:03,229 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0337be0f939be2a2b593eefecee32758: 2023-03-29 03:56:03,229 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758., storeName=0337be0f939be2a2b593eefecee32758/C, priority=12, startTime=1680062162714; duration=0sec 2023-03-29 03:56:03,229 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:56:03,229 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:03,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:03,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:03,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:03,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:03,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:03,231 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:03,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:03,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:03,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:03,233 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:03,233 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:03,233 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:03,233 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:03,233 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:56:03,233 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:56:03,233 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:03,234 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:03,234 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:03,234 DEBUG [Thread-3261] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:56:03,234 DEBUG [Thread-3261] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:56:03,233 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:03,236 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:03,236 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:03,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:03,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:03,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:03,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:03,334 DEBUG [Thread-3261] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:56:03,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:56:03,337 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:56:03,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:56:03,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:56:03,338 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:56:03,338 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:03,338 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:56:03,338 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:56:03,343 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:03,343 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:03,345 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:03,345 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:03,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:03,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:03,346 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:03,346 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:56:03,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:03,346 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:03,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:56:03,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:03,347 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:56:03,347 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:56:03,347 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:56:03,347 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:56:03,347 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:03,347 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:56:03,348 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:56:03,348 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:56:03,348 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:56:03,348 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. started... 2023-03-29 03:56:03,348 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. as already flushing 2023-03-29 03:56:03,348 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:56:03,348 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:56:03,348 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:56:03,348 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:03,348 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:56:03,348 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:56:03,350 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:03,350 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:03,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:03,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:03,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:03,350 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:03,350 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:56:03,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:03,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:03,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:03,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:03,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:03,353 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:56:03,353 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3250b7f3[Count = 0] remaining members to acquire global barrier 2023-03-29 03:56:03,353 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:56:03,353 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:03,355 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:03,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:03,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:03,355 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:56:03,355 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:03,355 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:56:03,355 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:56:03,355 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:56:03,356 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:56:03,356 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:03,356 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:56:03,357 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:03,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:03,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:03,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:03,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:03,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:03,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:03,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:03,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:03,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:03,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:03,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:56:03,360 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:56:03,360 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:56:03,360 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:56:03,360 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:56:03,360 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:56:03,360 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:56:03,362 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:03,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:03,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:03,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:03,362 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:03,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:03,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:03,362 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:03,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:03,363 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:03,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:03,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:03,363 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:03,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:03,363 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:03,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41942 deadline: 1680062223357, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:03,364 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:03,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 320 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42010 deadline: 1680062223358, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:03,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:03,365 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:03,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:03,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:03,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:03,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:03,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:03,367 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:03,367 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:03,367 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:03,367 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:03,367 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:03,368 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:03,368 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:56:03,368 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:56:03,367 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:03,368 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:03,368 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:03,368 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:03,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:03,368 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:56:03,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:03,368 DEBUG [Thread-3261] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:56:03,368 DEBUG [Thread-3261] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:56:03,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:03,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:03,374 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:03,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42024 deadline: 1680062223371, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:03,375 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:03,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 329 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42004 deadline: 1680062223371, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:03,442 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=654 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/A/655cd655a2714e678f9ce1310ef465f2 2023-03-29 03:56:03,455 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=654 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/B/4088eba980fd4881981cf6c282370dc5 2023-03-29 03:56:03,468 WARN [DataStreamer for file /user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/C/90677e947d784b5ab6b3c7f7a29f06a5] hdfs.DataStreamer(982): Caught exception java.lang.InterruptedException at java.lang.Object.wait(Native Method) at java.lang.Thread.join(Thread.java:1257) at java.lang.Thread.join(Thread.java:1331) at org.apache.hadoop.hdfs.DataStreamer.closeResponder(DataStreamer.java:980) at org.apache.hadoop.hdfs.DataStreamer.endBlock(DataStreamer.java:630) at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:807) 2023-03-29 03:56:03,473 DEBUG [Thread-3261] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:56:03,473 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=654 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/C/90677e947d784b5ab6b3c7f7a29f06a5 2023-03-29 03:56:03,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:56:03,475 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:56:03,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:56:03,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:56:03,478 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:56:03,478 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:03,478 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:56:03,478 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:56:03,480 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:03,480 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:03,481 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/A/655cd655a2714e678f9ce1310ef465f2 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/655cd655a2714e678f9ce1310ef465f2 2023-03-29 03:56:03,481 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:03,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:03,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:03,481 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:03,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:03,482 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:03,482 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:03,482 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:56:03,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:56:03,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:03,482 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:56:03,483 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:56:03,483 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:56:03,483 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:56:03,483 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:03,483 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:56:03,483 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:56:03,483 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:56:03,483 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:56:03,483 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. started... 2023-03-29 03:56:03,483 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. as already flushing 2023-03-29 03:56:03,483 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:56:03,484 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:56:03,484 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:56:03,484 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:03,484 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:56:03,484 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:56:03,487 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/655cd655a2714e678f9ce1310ef465f2, entries=200, sequenceid=654, filesize=14.3 K 2023-03-29 03:56:03,488 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/B/4088eba980fd4881981cf6c282370dc5 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/4088eba980fd4881981cf6c282370dc5 2023-03-29 03:56:03,488 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:03,488 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:03,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:03,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:03,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:03,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:03,488 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:03,488 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:56:03,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:03,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:03,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:03,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:03,490 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:56:03,490 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@71b606b5[Count = 0] remaining members to acquire global barrier 2023-03-29 03:56:03,490 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:56:03,490 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:03,492 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:03,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:03,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:03,492 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:56:03,492 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:56:03,492 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:56:03,493 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:03,493 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:56:03,494 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:56:03,494 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:03,494 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:56:03,494 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/4088eba980fd4881981cf6c282370dc5, entries=150, sequenceid=654, filesize=11.9 K 2023-03-29 03:56:03,494 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:03,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:03,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:03,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:03,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:03,495 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/C/90677e947d784b5ab6b3c7f7a29f06a5 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/90677e947d784b5ab6b3c7f7a29f06a5 2023-03-29 03:56:03,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:03,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:03,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:03,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:03,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:03,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:03,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:56:03,498 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:56:03,498 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:56:03,498 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:56:03,498 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:56:03,498 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:56:03,498 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:56:03,501 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/90677e947d784b5ab6b3c7f7a29f06a5, entries=150, sequenceid=654, filesize=11.9 K 2023-03-29 03:56:03,501 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:03,501 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:03,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:03,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:03,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:03,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:03,501 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:03,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:03,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:03,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:03,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:03,502 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~360 KB/368640, currentSize=53.67 KB/54960 for 0337be0f939be2a2b593eefecee32758 in 473ms, sequenceid=654, compaction requested=true 2023-03-29 03:56:03,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:03,502 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 0337be0f939be2a2b593eefecee32758: 2023-03-29 03:56:03,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:03,502 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:56:03,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:03,503 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-03-29 03:56:03,502 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:03,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:03,503 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:56:03,503 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:56:03,503 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-03-29 03:56:03,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:03,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:03,504 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-03-29 03:56:03,504 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-03-29 03:56:03,504 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. because compaction request was cancelled 2023-03-29 03:56:03,504 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-29 03:56:03,504 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-03-29 03:56:03,504 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-03-29 03:56:03,504 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. because compaction request was cancelled 2023-03-29 03:56:03,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:03,505 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:03,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:03,506 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47696 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-29 03:56:03,507 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1912): 0337be0f939be2a2b593eefecee32758/C is initiating minor compaction (all files) 2023-03-29 03:56:03,507 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0337be0f939be2a2b593eefecee32758/C in TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:56:03,507 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/109e3df5aa334d31b94d54b75a7f09cf, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/3db623b7a43d4f1c85f6275ca14b09b7, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/564227d153394044a02da50f51902990, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/90677e947d784b5ab6b3c7f7a29f06a5] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp, totalSize=46.6 K 2023-03-29 03:56:03,507 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 109e3df5aa334d31b94d54b75a7f09cf, keycount=150, bloomtype=ROW, size=13.3 K, encoding=NONE, compression=NONE, seqNum=594, earliestPutTs=1680062162322 2023-03-29 03:56:03,507 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 3db623b7a43d4f1c85f6275ca14b09b7, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=604, earliestPutTs=1680062162657 2023-03-29 03:56:03,508 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 564227d153394044a02da50f51902990, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=631, earliestPutTs=1680062162844 2023-03-29 03:56:03,508 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:03,508 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:03,508 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:03,508 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:03,508 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:03,508 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:03,508 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:56:03,508 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:56:03,509 DEBUG [Thread-3261] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:56:03,509 DEBUG [Thread-3261] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:56:03,508 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:03,508 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 90677e947d784b5ab6b3c7f7a29f06a5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=654, earliestPutTs=1680062162898 2023-03-29 03:56:03,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:03,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:03,508 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:03,510 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:03,510 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:03,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:03,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:03,534 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] throttle.PressureAwareThroughputController(145): 0337be0f939be2a2b593eefecee32758#C#compaction#564 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:56:03,609 DEBUG [Thread-3261] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:56:03,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:56:03,611 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:56:03,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:56:03,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:56:03,612 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:56:03,613 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:03,613 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:56:03,613 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:56:03,613 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:03,613 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:03,618 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:03,618 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:03,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:03,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:03,619 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:03,619 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:56:03,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:03,619 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:03,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:56:03,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:03,620 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:56:03,620 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:56:03,620 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:56:03,620 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:56:03,620 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:03,620 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:56:03,621 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:56:03,621 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:56:03,621 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:56:03,621 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. started... 2023-03-29 03:56:03,622 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0337be0f939be2a2b593eefecee32758 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-29 03:56:03,622 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0337be0f939be2a2b593eefecee32758, store=A 2023-03-29 03:56:03,622 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:56:03,622 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0337be0f939be2a2b593eefecee32758, store=B 2023-03-29 03:56:03,622 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:56:03,622 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0337be0f939be2a2b593eefecee32758, store=C 2023-03-29 03:56:03,622 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:56:03,645 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=666 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/A/50d00105bf4848c1b04809f888c36e56 2023-03-29 03:56:03,666 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=666 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/B/8ec7a596d0994c529676ff038ba7fe84 2023-03-29 03:56:03,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on 0337be0f939be2a2b593eefecee32758 2023-03-29 03:56:03,669 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. as already flushing 2023-03-29 03:56:03,685 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=666 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/C/2b65898d1a6b4249a1087f8fea67a104 2023-03-29 03:56:03,688 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/A/50d00105bf4848c1b04809f888c36e56 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/50d00105bf4848c1b04809f888c36e56 2023-03-29 03:56:03,691 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/50d00105bf4848c1b04809f888c36e56, entries=150, sequenceid=666, filesize=11.9 K 2023-03-29 03:56:03,692 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/B/8ec7a596d0994c529676ff038ba7fe84 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/8ec7a596d0994c529676ff038ba7fe84 2023-03-29 03:56:03,695 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/8ec7a596d0994c529676ff038ba7fe84, entries=150, sequenceid=666, filesize=11.9 K 2023-03-29 03:56:03,696 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/C/2b65898d1a6b4249a1087f8fea67a104 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/2b65898d1a6b4249a1087f8fea67a104 2023-03-29 03:56:03,705 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/2b65898d1a6b4249a1087f8fea67a104, entries=150, sequenceid=666, filesize=11.9 K 2023-03-29 03:56:03,706 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=53.67 KB/54960 for 0337be0f939be2a2b593eefecee32758 in 85ms, sequenceid=666, compaction requested=true 2023-03-29 03:56:03,706 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0337be0f939be2a2b593eefecee32758: 2023-03-29 03:56:03,706 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:56:03,706 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:56:03,706 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:56:03,706 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:03,706 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:56:03,706 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:56:03,708 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:03,708 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:03,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:03,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:03,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:03,709 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:03,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:03,709 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:56:03,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:03,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:03,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:03,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:03,710 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:56:03,710 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6b92418[Count = 0] remaining members to acquire global barrier 2023-03-29 03:56:03,710 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:56:03,710 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:03,711 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:03,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:03,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:03,712 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:56:03,712 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:56:03,712 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:56:03,712 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:03,712 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:56:03,714 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:03,714 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:56:03,714 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:03,714 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:56:03,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:03,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:03,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:03,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:03,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:03,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:03,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on 0337be0f939be2a2b593eefecee32758 2023-03-29 03:56:03,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:03,716 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 0337be0f939be2a2b593eefecee32758 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-29 03:56:03,716 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0337be0f939be2a2b593eefecee32758, store=A 2023-03-29 03:56:03,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:03,716 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:56:03,716 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0337be0f939be2a2b593eefecee32758, store=B 2023-03-29 03:56:03,717 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:56:03,717 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0337be0f939be2a2b593eefecee32758, store=C 2023-03-29 03:56:03,717 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:56:03,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:03,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:03,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:56:03,720 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:56:03,720 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:56:03,720 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:56:03,720 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:56:03,720 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:56:03,720 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:56:03,726 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:03,726 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:03,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:03,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:03,726 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:03,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:03,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:03,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:03,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:03,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:03,727 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=678 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/A/66a52fd08fa440ea945ccef1c996eee1 2023-03-29 03:56:03,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:03,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:03,728 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:03,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:03,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:03,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:03,730 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:03,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:03,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:03,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:03,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:03,734 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:03,734 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:03,734 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:03,734 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:03,734 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:03,734 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:03,734 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:03,735 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:03,735 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:56:03,735 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:03,735 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:56:03,735 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:56:03,735 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:03,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:03,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:03,735 DEBUG [Thread-3261] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:56:03,735 DEBUG [Thread-3261] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:56:03,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:03,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:03,749 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=678 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/B/957a9a99004145f6af5d3147ad15f88e 2023-03-29 03:56:03,755 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:03,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 329 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42010 deadline: 1680062223752, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:03,762 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:03,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 336 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42004 deadline: 1680062223753, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:03,764 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=678 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/C/8083af365b3049a6b5b315ee7139b77d 2023-03-29 03:56:03,764 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:03,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41942 deadline: 1680062223754, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:03,764 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:03,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42024 deadline: 1680062223755, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:03,767 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/A/66a52fd08fa440ea945ccef1c996eee1 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/66a52fd08fa440ea945ccef1c996eee1 2023-03-29 03:56:03,770 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/66a52fd08fa440ea945ccef1c996eee1, entries=200, sequenceid=678, filesize=14.3 K 2023-03-29 03:56:03,770 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/B/957a9a99004145f6af5d3147ad15f88e as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/957a9a99004145f6af5d3147ad15f88e 2023-03-29 03:56:03,773 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/957a9a99004145f6af5d3147ad15f88e, entries=150, sequenceid=678, filesize=11.9 K 2023-03-29 03:56:03,776 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/C/8083af365b3049a6b5b315ee7139b77d as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/8083af365b3049a6b5b315ee7139b77d 2023-03-29 03:56:03,779 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/8083af365b3049a6b5b315ee7139b77d, entries=150, sequenceid=678, filesize=11.9 K 2023-03-29 03:56:03,779 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 0337be0f939be2a2b593eefecee32758 in 63ms, sequenceid=678, compaction requested=true 2023-03-29 03:56:03,779 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 0337be0f939be2a2b593eefecee32758: 2023-03-29 03:56:03,780 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:56:03,780 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:56:03,780 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-29 03:56:03,780 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-29 03:56:03,781 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 55122 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-29 03:56:03,781 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1912): 0337be0f939be2a2b593eefecee32758/A is initiating minor compaction (all files) 2023-03-29 03:56:03,781 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 0337be0f939be2a2b593eefecee32758/A in TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:56:03,781 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/5d72012d115b4fa094ec80ef59eed0eb, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/655cd655a2714e678f9ce1310ef465f2, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/50d00105bf4848c1b04809f888c36e56, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/66a52fd08fa440ea945ccef1c996eee1] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp, totalSize=53.8 K 2023-03-29 03:56:03,781 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 5d72012d115b4fa094ec80ef59eed0eb, keycount=150, bloomtype=ROW, size=13.4 K, encoding=NONE, compression=NONE, seqNum=631, earliestPutTs=1680062162844 2023-03-29 03:56:03,783 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 655cd655a2714e678f9ce1310ef465f2, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=654, earliestPutTs=1680062162898 2023-03-29 03:56:03,783 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 50d00105bf4848c1b04809f888c36e56, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=666, earliestPutTs=1680062163037 2023-03-29 03:56:03,784 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 66a52fd08fa440ea945ccef1c996eee1, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=678, earliestPutTs=1680062163683 2023-03-29 03:56:03,793 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0337be0f939be2a2b593eefecee32758#A#compaction#571 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:56:03,803 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/A/f658f3ea5f51467d83cc627d02bdc291 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/f658f3ea5f51467d83cc627d02bdc291 2023-03-29 03:56:03,810 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 0337be0f939be2a2b593eefecee32758/A of 0337be0f939be2a2b593eefecee32758 into f658f3ea5f51467d83cc627d02bdc291(size=13.5 K), total size for store is 13.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:56:03,810 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0337be0f939be2a2b593eefecee32758: 2023-03-29 03:56:03,810 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758., storeName=0337be0f939be2a2b593eefecee32758/A, priority=12, startTime=1680062163780; duration=0sec 2023-03-29 03:56:03,810 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-29 03:56:03,810 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 4 compacting, 2 eligible, 16 blocking 2023-03-29 03:56:03,811 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-03-29 03:56:03,811 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-03-29 03:56:03,811 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. because compaction request was cancelled 2023-03-29 03:56:03,811 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-29 03:56:03,812 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 50242 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-29 03:56:03,812 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1912): 0337be0f939be2a2b593eefecee32758/B is initiating minor compaction (all files) 2023-03-29 03:56:03,812 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 0337be0f939be2a2b593eefecee32758/B in TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:56:03,812 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/13a6ca1c587a49a2b16109f0e4e8a872, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/4088eba980fd4881981cf6c282370dc5, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/8ec7a596d0994c529676ff038ba7fe84, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/957a9a99004145f6af5d3147ad15f88e] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp, totalSize=49.1 K 2023-03-29 03:56:03,813 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 13a6ca1c587a49a2b16109f0e4e8a872, keycount=150, bloomtype=ROW, size=13.4 K, encoding=NONE, compression=NONE, seqNum=631, earliestPutTs=1680062162844 2023-03-29 03:56:03,813 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 4088eba980fd4881981cf6c282370dc5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=654, earliestPutTs=1680062162898 2023-03-29 03:56:03,813 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 8ec7a596d0994c529676ff038ba7fe84, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=666, earliestPutTs=1680062163037 2023-03-29 03:56:03,813 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 957a9a99004145f6af5d3147ad15f88e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=678, earliestPutTs=1680062163688 2023-03-29 03:56:03,820 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0337be0f939be2a2b593eefecee32758#B#compaction#572 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:56:03,829 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/B/2ee6ec5e22064976bb14b0af3a4f5631 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/2ee6ec5e22064976bb14b0af3a4f5631 2023-03-29 03:56:03,835 DEBUG [Thread-3261] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:56:03,835 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 0337be0f939be2a2b593eefecee32758/B of 0337be0f939be2a2b593eefecee32758 into 2ee6ec5e22064976bb14b0af3a4f5631(size=13.5 K), total size for store is 13.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:56:03,836 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0337be0f939be2a2b593eefecee32758: 2023-03-29 03:56:03,836 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758., storeName=0337be0f939be2a2b593eefecee32758/B, priority=12, startTime=1680062163780; duration=0sec 2023-03-29 03:56:03,836 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:56:03,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:56:03,838 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:56:03,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:56:03,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:56:03,839 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:56:03,839 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:03,839 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:56:03,839 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:56:03,839 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:03,839 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:03,841 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:03,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:03,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:03,841 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:03,841 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:03,842 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:56:03,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:03,843 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:03,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:56:03,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:03,843 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:56:03,843 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:56:03,843 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:56:03,844 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:56:03,844 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:03,844 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:56:03,852 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:56:03,852 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:56:03,852 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. started... 2023-03-29 03:56:03,852 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:56:03,852 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0337be0f939be2a2b593eefecee32758 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-29 03:56:03,852 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0337be0f939be2a2b593eefecee32758, store=A 2023-03-29 03:56:03,852 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:56:03,852 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0337be0f939be2a2b593eefecee32758, store=B 2023-03-29 03:56:03,852 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:56:03,852 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0337be0f939be2a2b593eefecee32758, store=C 2023-03-29 03:56:03,853 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:56:03,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on 0337be0f939be2a2b593eefecee32758 2023-03-29 03:56:03,861 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. as already flushing 2023-03-29 03:56:03,871 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=700 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/A/b009847b456c439387422ef49d1987e1 2023-03-29 03:56:03,882 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:03,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 334 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42010 deadline: 1680062223874, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:03,884 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:03,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 340 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42004 deadline: 1680062223875, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:03,885 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:03,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42024 deadline: 1680062223878, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:03,886 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:03,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41942 deadline: 1680062223879, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:03,887 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=700 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/B/f2a825f4ff7b484088f9f817dee7c8f0 2023-03-29 03:56:03,901 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=700 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/C/d720438a74b843edb3f074a46809bcdb 2023-03-29 03:56:03,905 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/A/b009847b456c439387422ef49d1987e1 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/b009847b456c439387422ef49d1987e1 2023-03-29 03:56:03,910 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/b009847b456c439387422ef49d1987e1, entries=150, sequenceid=700, filesize=11.9 K 2023-03-29 03:56:03,910 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/B/f2a825f4ff7b484088f9f817dee7c8f0 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/f2a825f4ff7b484088f9f817dee7c8f0 2023-03-29 03:56:03,917 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/f2a825f4ff7b484088f9f817dee7c8f0, entries=150, sequenceid=700, filesize=11.9 K 2023-03-29 03:56:03,918 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/C/d720438a74b843edb3f074a46809bcdb as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/d720438a74b843edb3f074a46809bcdb 2023-03-29 03:56:03,921 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/d720438a74b843edb3f074a46809bcdb, entries=150, sequenceid=700, filesize=11.9 K 2023-03-29 03:56:03,921 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 0337be0f939be2a2b593eefecee32758 in 69ms, sequenceid=700, compaction requested=true 2023-03-29 03:56:03,922 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0337be0f939be2a2b593eefecee32758: 2023-03-29 03:56:03,922 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:56:03,922 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:56:03,922 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:56:03,922 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:03,922 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:56:03,922 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:56:03,927 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:03,927 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:03,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:03,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:03,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:03,927 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:03,927 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:56:03,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:03,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:03,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:03,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:03,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:03,929 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:56:03,929 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3901e86d[Count = 0] remaining members to acquire global barrier 2023-03-29 03:56:03,930 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:56:03,930 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:03,931 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:03,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:03,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:03,931 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:56:03,931 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:56:03,931 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:56:03,932 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:03,932 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:56:03,933 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:03,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:03,933 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:56:03,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:03,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:03,933 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:03,933 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:56:03,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:03,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:03,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:03,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:03,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:03,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:03,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:03,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:56:03,936 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:56:03,936 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:56:03,936 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:56:03,937 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:56:03,937 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:56:03,937 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:56:03,938 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:03,938 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:03,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:03,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:03,938 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:03,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:03,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:03,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:03,939 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:03,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:03,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:03,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:03,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:03,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:03,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:03,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:03,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:03,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:03,941 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:03,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:03,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:03,945 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:03,945 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:03,945 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:03,945 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:03,945 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:56:03,945 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:56:03,946 DEBUG [Thread-3261] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:56:03,946 DEBUG [Thread-3261] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:56:03,945 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:03,946 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:03,946 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:03,946 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:03,946 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:03,946 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:03,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:03,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:03,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:03,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:03,948 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/C/d28603b66bb7487a89adf3dd518c380d as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/d28603b66bb7487a89adf3dd518c380d 2023-03-29 03:56:03,953 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 0337be0f939be2a2b593eefecee32758/C of 0337be0f939be2a2b593eefecee32758 into d28603b66bb7487a89adf3dd518c380d(size=13.4 K), total size for store is 49.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:56:03,953 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0337be0f939be2a2b593eefecee32758: 2023-03-29 03:56:03,953 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758., storeName=0337be0f939be2a2b593eefecee32758/C, priority=12, startTime=1680062163503; duration=0sec 2023-03-29 03:56:03,953 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:56:03,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on 0337be0f939be2a2b593eefecee32758 2023-03-29 03:56:03,986 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 0337be0f939be2a2b593eefecee32758 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-29 03:56:03,986 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0337be0f939be2a2b593eefecee32758, store=A 2023-03-29 03:56:03,986 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:56:03,986 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0337be0f939be2a2b593eefecee32758, store=B 2023-03-29 03:56:03,986 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:56:03,987 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0337be0f939be2a2b593eefecee32758, store=C 2023-03-29 03:56:03,987 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:56:04,002 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=714 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/A/42a66d4e8a1347199c27b67c55cfa960 2023-03-29 03:56:04,033 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:04,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41942 deadline: 1680062224027, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:04,034 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:04,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42024 deadline: 1680062224028, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:04,039 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:04,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 341 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42010 deadline: 1680062224029, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:04,039 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:04,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 347 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42004 deadline: 1680062224029, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:04,046 DEBUG [Thread-3261] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:56:04,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:56:04,048 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:56:04,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:56:04,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:56:04,049 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:56:04,049 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:04,049 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:56:04,049 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:56:04,050 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:04,050 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:04,052 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:04,052 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:04,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:04,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:04,053 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:04,053 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:56:04,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:04,053 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:04,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:56:04,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:04,054 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:56:04,054 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:56:04,054 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:56:04,054 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:56:04,054 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:04,054 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:56:04,055 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:56:04,055 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:56:04,056 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:56:04,056 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. started... 2023-03-29 03:56:04,056 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. as already flushing 2023-03-29 03:56:04,056 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:56:04,056 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:56:04,056 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:56:04,056 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:04,056 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:56:04,056 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:56:04,058 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:04,058 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:04,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:04,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:04,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:04,058 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:04,058 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:56:04,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:04,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:04,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:04,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:04,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:04,060 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:56:04,060 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ed93f91[Count = 0] remaining members to acquire global barrier 2023-03-29 03:56:04,060 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:56:04,060 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:04,062 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:04,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:04,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:04,063 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:56:04,063 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:56:04,063 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:56:04,063 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:04,063 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:56:04,065 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:04,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:04,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:04,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:04,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:04,066 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:56:04,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:04,066 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:04,066 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:56:04,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:04,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:04,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:04,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:04,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:04,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:56:04,068 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:56:04,068 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:56:04,068 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:56:04,068 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:56:04,068 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:56:04,068 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:56:04,070 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:04,070 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:04,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:04,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:04,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:04,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:04,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:04,070 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:04,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:04,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:04,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:04,071 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:04,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:04,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:04,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:04,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:04,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:04,072 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:04,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:04,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:04,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:04,077 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:04,077 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:04,077 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:04,077 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:04,077 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:04,078 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:56:04,078 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:04,078 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:56:04,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:04,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:04,078 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:04,078 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:04,078 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:04,078 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:56:04,078 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:04,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:04,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:04,078 DEBUG [Thread-3261] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:56:04,078 DEBUG [Thread-3261] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:56:04,137 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:04,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41942 deadline: 1680062224134, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:04,138 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:04,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42024 deadline: 1680062224135, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:04,147 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:04,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 343 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42010 deadline: 1680062224140, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:04,148 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:04,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 349 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42004 deadline: 1680062224140, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:04,178 DEBUG [Thread-3261] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:56:04,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:56:04,181 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:56:04,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:56:04,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:56:04,182 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:56:04,182 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:04,182 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:56:04,182 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:56:04,182 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:04,182 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:04,184 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:04,184 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:04,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:04,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:04,184 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:04,184 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:56:04,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:04,185 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:04,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:56:04,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:04,185 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:56:04,185 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:56:04,185 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:56:04,186 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:56:04,186 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:04,186 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:56:04,186 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:56:04,186 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:56:04,186 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:56:04,186 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. started... 2023-03-29 03:56:04,186 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. as already flushing 2023-03-29 03:56:04,186 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:56:04,186 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:56:04,186 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:56:04,186 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:04,186 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:56:04,186 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:56:04,189 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:04,189 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:04,189 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:04,189 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:56:04,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:04,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:04,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:04,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:04,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:04,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:04,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:04,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:04,193 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:56:04,193 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f1b0351[Count = 0] remaining members to acquire global barrier 2023-03-29 03:56:04,193 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:56:04,193 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:04,202 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:04,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:04,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:04,202 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:56:04,202 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:56:04,202 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:56:04,202 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:04,202 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:56:04,204 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:04,204 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:56:04,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:04,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:04,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:04,204 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:04,204 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:56:04,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:04,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:04,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:04,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:04,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:04,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:04,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:04,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:56:04,206 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:56:04,206 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:56:04,206 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:56:04,206 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:56:04,206 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:56:04,206 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:56:04,209 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:04,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:04,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:04,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:04,209 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:04,209 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:04,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:04,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:04,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:04,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:04,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:04,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:04,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:04,210 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:04,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:04,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:04,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:04,211 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:04,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:04,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:04,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:04,216 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:04,216 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:04,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:04,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:04,217 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:04,217 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:56:04,217 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:56:04,217 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:04,217 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:04,217 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:04,217 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:04,217 DEBUG [Thread-3261] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:56:04,217 DEBUG [Thread-3261] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:56:04,217 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:04,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:04,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:04,217 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:04,217 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:04,317 DEBUG [Thread-3261] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:56:04,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:56:04,320 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:56:04,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:56:04,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:56:04,321 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:56:04,321 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:04,321 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:56:04,321 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:56:04,328 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:04,328 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:04,330 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:04,330 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:04,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:04,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:04,330 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:04,330 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:56:04,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:04,331 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:04,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:56:04,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:04,331 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:56:04,331 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:56:04,331 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:56:04,331 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:56:04,331 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:04,332 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:56:04,332 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:56:04,332 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:56:04,332 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:56:04,332 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. started... 2023-03-29 03:56:04,332 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. as already flushing 2023-03-29 03:56:04,332 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:56:04,332 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:56:04,332 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:56:04,332 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:04,332 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:56:04,332 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:56:04,334 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:04,334 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:04,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:04,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:04,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:04,335 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:04,335 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:56:04,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:04,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:04,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:04,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:04,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:04,336 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:56:04,336 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@cd674e[Count = 0] remaining members to acquire global barrier 2023-03-29 03:56:04,336 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:56:04,336 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:04,338 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:04,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:04,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:04,338 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:56:04,338 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:56:04,338 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:56:04,338 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:04,338 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:56:04,340 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:04,340 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:56:04,340 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:04,340 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:56:04,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:04,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:04,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:04,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:04,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:04,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:04,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:04,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:04,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:04,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:04,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:56:04,343 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:56:04,343 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:56:04,343 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:56:04,343 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:56:04,343 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:56:04,343 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:56:04,345 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:04,345 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:04,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:04,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:04,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:04,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:04,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:04,345 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:04,345 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:04,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:04,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41942 deadline: 1680062224338, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:04,346 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:04,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:04,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42024 deadline: 1680062224340, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:04,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:04,346 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:04,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:04,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:04,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:04,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:04,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:04,348 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:04,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:04,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:04,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:04,352 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:04,352 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:04,352 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:04,352 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:04,352 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:04,352 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:04,352 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:04,352 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:04,352 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:56:04,352 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:56:04,352 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:04,352 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:04,352 DEBUG [Thread-3261] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:56:04,353 DEBUG [Thread-3261] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:56:04,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:04,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:04,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:04,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:04,354 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:04,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 351 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42004 deadline: 1680062224349, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:04,356 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:04,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 345 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42010 deadline: 1680062224349, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:04,415 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=714 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/B/2aa398b21223491a89876fa170a34762 2023-03-29 03:56:04,453 DEBUG [Thread-3261] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:56:04,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:56:04,455 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:56:04,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:56:04,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:56:04,456 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:56:04,456 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:04,456 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:56:04,456 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:56:04,457 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:04,457 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:04,458 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:04,458 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:04,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:04,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:04,458 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:04,458 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:56:04,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:04,459 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:04,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:56:04,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:04,459 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:56:04,459 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:56:04,459 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:56:04,460 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:56:04,460 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:04,460 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:56:04,460 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:56:04,460 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:56:04,460 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:56:04,460 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. started... 2023-03-29 03:56:04,460 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. as already flushing 2023-03-29 03:56:04,460 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:56:04,460 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:56:04,460 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:56:04,460 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:04,460 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:56:04,461 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:56:04,464 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:04,464 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:04,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:04,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:04,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:04,464 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:04,464 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:56:04,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:04,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:04,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:04,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:04,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:04,466 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:56:04,466 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3639b5ba[Count = 0] remaining members to acquire global barrier 2023-03-29 03:56:04,466 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:56:04,466 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:04,467 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:04,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:04,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:04,467 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:56:04,467 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:56:04,467 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:56:04,467 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:04,467 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:56:04,468 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:04,469 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:56:04,469 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:04,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:04,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:04,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:04,469 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:56:04,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:04,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:04,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:04,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:04,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:04,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:04,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:04,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:56:04,476 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:56:04,476 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:56:04,476 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:56:04,476 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:56:04,476 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:56:04,476 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:56:04,478 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:04,478 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:04,478 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:04,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:04,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:04,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:04,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:04,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:04,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:04,478 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:04,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:04,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:04,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:04,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:04,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:04,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:04,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:04,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:04,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:04,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:04,483 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:04,486 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:04,486 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:04,486 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:04,486 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:04,486 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:56:04,486 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:04,486 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:56:04,486 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:56:04,486 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:04,486 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:04,486 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:04,486 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:04,486 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:04,486 DEBUG [Thread-3261] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:56:04,486 DEBUG [Thread-3261] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:56:04,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:04,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:04,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:04,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:04,586 DEBUG [Thread-3261] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:56:04,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:56:04,589 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:56:04,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:56:04,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:56:04,590 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:56:04,590 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:04,590 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:56:04,590 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:56:04,590 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:04,591 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:04,592 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:04,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:04,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:04,592 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:04,592 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:04,592 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:56:04,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:04,593 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:04,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:56:04,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:04,593 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:56:04,593 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:56:04,593 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:56:04,593 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:56:04,594 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:04,594 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:56:04,594 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:56:04,594 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:56:04,594 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:56:04,594 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. started... 2023-03-29 03:56:04,594 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. as already flushing 2023-03-29 03:56:04,594 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:56:04,594 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:56:04,594 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:56:04,594 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:04,594 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:56:04,594 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:56:04,596 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:04,596 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:04,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:04,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:04,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:04,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:04,598 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:04,598 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:56:04,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:04,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:04,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:04,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:04,599 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:56:04,599 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@715b215b[Count = 0] remaining members to acquire global barrier 2023-03-29 03:56:04,599 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:56:04,599 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:04,600 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:04,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:04,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:04,601 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:56:04,601 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:56:04,601 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:56:04,601 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:04,601 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:56:04,606 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:04,606 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:56:04,606 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:04,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:04,606 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:56:04,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:04,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:04,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:04,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:04,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:04,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:04,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:04,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:04,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:04,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:56:04,609 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:56:04,609 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:56:04,609 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:56:04,609 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:56:04,609 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:56:04,609 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:56:04,611 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:04,611 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:04,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:04,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:04,612 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:04,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:04,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:04,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:04,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:04,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:04,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:04,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:04,613 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:04,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:04,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:04,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:04,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:04,614 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:04,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:04,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:04,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:04,617 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:04,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41960 deadline: 1680062224610, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:04,618 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:04,618 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:04,618 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:04,618 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:04,618 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:04,618 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:04,618 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:04,620 DEBUG [Thread-3251] client.RpcRetryingCallerImpl(129): Call exception, tries=8, retries=16, started=18320 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758., hostname=jenkins-hbase4.apache.org,35599,1680062018534, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-29 03:56:04,620 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:56:04,620 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:56:04,620 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:04,620 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:04,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:04,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:04,620 DEBUG [Thread-3261] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:56:04,620 DEBUG [Thread-3261] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:56:04,620 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:04,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:04,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:04,650 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:04,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41942 deadline: 1680062224647, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:04,650 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:04,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42024 deadline: 1680062224648, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:04,659 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:04,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 353 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42004 deadline: 1680062224657, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:04,659 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:04,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 347 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:42010 deadline: 1680062224657, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:04,720 DEBUG [Thread-3261] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:56:04,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:56:04,723 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:56:04,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:56:04,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:56:04,724 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:56:04,724 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:04,724 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:56:04,724 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:56:04,724 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:04,725 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:04,727 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:04,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:04,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:04,727 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:04,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:04,728 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:04,728 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:56:04,728 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:04,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:56:04,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:04,728 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:56:04,729 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:56:04,729 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:56:04,729 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:56:04,729 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:04,729 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:56:04,732 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:56:04,732 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:56:04,732 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:56:04,732 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. started... 2023-03-29 03:56:04,732 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. as already flushing 2023-03-29 03:56:04,732 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:56:04,732 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:56:04,732 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:56:04,732 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:04,732 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:56:04,732 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:56:04,734 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:04,734 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:04,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:04,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:04,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:04,734 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:04,734 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:56:04,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:04,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:04,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:04,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:04,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:04,736 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:56:04,736 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@664da5aa[Count = 0] remaining members to acquire global barrier 2023-03-29 03:56:04,736 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:56:04,736 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:04,737 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:04,737 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:04,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:04,737 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:56:04,737 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:56:04,737 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:56:04,741 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:56:04,741 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:04,741 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:56:04,741 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:04,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:04,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:04,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:04,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:04,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:04,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:04,742 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:04,742 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:56:04,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:04,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:04,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:04,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:04,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:56:04,744 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:56:04,744 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:56:04,744 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:56:04,744 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:56:04,744 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:56:04,744 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:56:04,746 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:04,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:04,747 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:04,747 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:04,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:04,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:04,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:04,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:04,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:04,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:04,747 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:04,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:04,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:04,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:04,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:04,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:04,749 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:04,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:04,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:04,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:04,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:04,751 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:04,751 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:04,751 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:04,751 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:04,751 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:56:04,751 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:56:04,751 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:04,752 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:04,752 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:04,752 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:04,751 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:04,752 DEBUG [Thread-3261] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:56:04,751 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:56:04,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:04,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:04,751 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:04,752 DEBUG [Thread-3261] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:56:04,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:04,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:04,825 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=714 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/C/184a069a2097447d8fb16d1485190ddd 2023-03-29 03:56:04,831 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/A/42a66d4e8a1347199c27b67c55cfa960 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/42a66d4e8a1347199c27b67c55cfa960 2023-03-29 03:56:04,835 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/42a66d4e8a1347199c27b67c55cfa960, entries=200, sequenceid=714, filesize=14.3 K 2023-03-29 03:56:04,835 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/B/2aa398b21223491a89876fa170a34762 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/2aa398b21223491a89876fa170a34762 2023-03-29 03:56:04,841 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/2aa398b21223491a89876fa170a34762, entries=150, sequenceid=714, filesize=11.9 K 2023-03-29 03:56:04,842 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/C/184a069a2097447d8fb16d1485190ddd as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/184a069a2097447d8fb16d1485190ddd 2023-03-29 03:56:04,846 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/184a069a2097447d8fb16d1485190ddd, entries=150, sequenceid=714, filesize=11.9 K 2023-03-29 03:56:04,846 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=114.05 KB/116790 for 0337be0f939be2a2b593eefecee32758 in 860ms, sequenceid=714, compaction requested=true 2023-03-29 03:56:04,846 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 0337be0f939be2a2b593eefecee32758: 2023-03-29 03:56:04,847 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:56:04,847 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-29 03:56:04,847 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:56:04,847 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-29 03:56:04,847 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-29 03:56:04,848 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 40632 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-29 03:56:04,848 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1912): 0337be0f939be2a2b593eefecee32758/A is initiating minor compaction (all files) 2023-03-29 03:56:04,848 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 0337be0f939be2a2b593eefecee32758/A in TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:56:04,848 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/f658f3ea5f51467d83cc627d02bdc291, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/b009847b456c439387422ef49d1987e1, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/42a66d4e8a1347199c27b67c55cfa960] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp, totalSize=39.7 K 2023-03-29 03:56:04,849 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting f658f3ea5f51467d83cc627d02bdc291, keycount=150, bloomtype=ROW, size=13.5 K, encoding=NONE, compression=NONE, seqNum=678, earliestPutTs=1680062163688 2023-03-29 03:56:04,850 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting b009847b456c439387422ef49d1987e1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=700, earliestPutTs=1680062163751 2023-03-29 03:56:04,850 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 62462 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-29 03:56:04,851 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1912): 0337be0f939be2a2b593eefecee32758/C is initiating minor compaction (all files) 2023-03-29 03:56:04,851 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0337be0f939be2a2b593eefecee32758/C in TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:56:04,851 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/d28603b66bb7487a89adf3dd518c380d, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/2b65898d1a6b4249a1087f8fea67a104, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/8083af365b3049a6b5b315ee7139b77d, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/d720438a74b843edb3f074a46809bcdb, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/184a069a2097447d8fb16d1485190ddd] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp, totalSize=61.0 K 2023-03-29 03:56:04,851 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 42a66d4e8a1347199c27b67c55cfa960, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=714, earliestPutTs=1680062163874 2023-03-29 03:56:04,851 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting d28603b66bb7487a89adf3dd518c380d, keycount=150, bloomtype=ROW, size=13.4 K, encoding=NONE, compression=NONE, seqNum=654, earliestPutTs=1680062162898 2023-03-29 03:56:04,852 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 2b65898d1a6b4249a1087f8fea67a104, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=666, earliestPutTs=1680062163037 2023-03-29 03:56:04,852 DEBUG [Thread-3261] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:56:04,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:56:04,853 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 8083af365b3049a6b5b315ee7139b77d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=678, earliestPutTs=1680062163688 2023-03-29 03:56:04,853 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting d720438a74b843edb3f074a46809bcdb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=700, earliestPutTs=1680062163751 2023-03-29 03:56:04,853 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 184a069a2097447d8fb16d1485190ddd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=714, earliestPutTs=1680062163874 2023-03-29 03:56:04,854 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:56:04,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:56:04,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:56:04,856 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:56:04,856 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:04,856 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:56:04,856 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:56:04,856 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:04,856 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:04,858 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:04,858 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:04,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:04,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:04,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:04,860 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:04,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:56:04,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:04,860 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:56:04,860 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:04,860 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:56:04,868 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:56:04,868 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:56:04,868 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:56:04,868 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:04,868 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:56:04,874 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] throttle.PressureAwareThroughputController(145): 0337be0f939be2a2b593eefecee32758#C#compaction#579 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:56:04,880 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:56:04,880 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:56:04,880 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:56:04,880 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. started... 2023-03-29 03:56:04,880 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 0337be0f939be2a2b593eefecee32758 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-29 03:56:04,881 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0337be0f939be2a2b593eefecee32758, store=A 2023-03-29 03:56:04,881 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:56:04,881 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0337be0f939be2a2b593eefecee32758, store=B 2023-03-29 03:56:04,881 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:56:04,881 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0337be0f939be2a2b593eefecee32758, store=C 2023-03-29 03:56:04,881 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:56:04,881 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] throttle.PressureAwareThroughputController(145): 0337be0f939be2a2b593eefecee32758#A#compaction#580 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:56:04,906 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/C/67aeeed84f094825ad66c35be23e19a0 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/67aeeed84f094825ad66c35be23e19a0 2023-03-29 03:56:04,913 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 0337be0f939be2a2b593eefecee32758/C of 0337be0f939be2a2b593eefecee32758 into 67aeeed84f094825ad66c35be23e19a0(size=13.6 K), total size for store is 13.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:56:04,913 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0337be0f939be2a2b593eefecee32758: 2023-03-29 03:56:04,913 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758., storeName=0337be0f939be2a2b593eefecee32758/C, priority=11, startTime=1680062164847; duration=0sec 2023-03-29 03:56:04,913 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:56:04,913 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-29 03:56:04,914 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 38192 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-29 03:56:04,914 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1912): 0337be0f939be2a2b593eefecee32758/B is initiating minor compaction (all files) 2023-03-29 03:56:04,914 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 0337be0f939be2a2b593eefecee32758/B in TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:56:04,914 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/2ee6ec5e22064976bb14b0af3a4f5631, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/f2a825f4ff7b484088f9f817dee7c8f0, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/2aa398b21223491a89876fa170a34762] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp, totalSize=37.3 K 2023-03-29 03:56:04,915 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 2ee6ec5e22064976bb14b0af3a4f5631, keycount=150, bloomtype=ROW, size=13.5 K, encoding=NONE, compression=NONE, seqNum=678, earliestPutTs=1680062163688 2023-03-29 03:56:04,915 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting f2a825f4ff7b484088f9f817dee7c8f0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=700, earliestPutTs=1680062163751 2023-03-29 03:56:04,915 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 2aa398b21223491a89876fa170a34762, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=714, earliestPutTs=1680062163874 2023-03-29 03:56:04,922 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] throttle.PressureAwareThroughputController(145): 0337be0f939be2a2b593eefecee32758#B#compaction#582 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:56:04,928 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=734 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/A/9aced92a98c64f1f96abcde50aa2b7c9 2023-03-29 03:56:05,124 DEBUG [Thread-3268] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3a9f3972 to 127.0.0.1:52103 2023-03-29 03:56:05,124 DEBUG [Thread-3268] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-29 03:56:05,125 DEBUG [Thread-3264] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6ed183a4 to 127.0.0.1:52103 2023-03-29 03:56:05,125 DEBUG [Thread-3264] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-29 03:56:05,127 DEBUG [Thread-3266] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x10a344e6 to 127.0.0.1:52103 2023-03-29 03:56:05,127 DEBUG [Thread-3266] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-29 03:56:05,129 DEBUG [Thread-3270] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2d9fccbf to 127.0.0.1:52103 2023-03-29 03:56:05,129 DEBUG [Thread-3262] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x549bef45 to 127.0.0.1:52103 2023-03-29 03:56:05,129 DEBUG [Thread-3270] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-29 03:56:05,129 DEBUG [Thread-3262] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-29 03:56:05,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on 0337be0f939be2a2b593eefecee32758 2023-03-29 03:56:05,153 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. as already flushing 2023-03-29 03:56:05,153 DEBUG [Thread-3253] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6f2b3c03 to 127.0.0.1:52103 2023-03-29 03:56:05,153 DEBUG [Thread-3253] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-29 03:56:05,154 DEBUG [Thread-3257] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x067ea274 to 127.0.0.1:52103 2023-03-29 03:56:05,154 DEBUG [Thread-3257] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-29 03:56:05,161 DEBUG [Thread-3255] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x70e2be04 to 127.0.0.1:52103 2023-03-29 03:56:05,162 DEBUG [Thread-3255] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-29 03:56:05,164 DEBUG [Thread-3259] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x78592df0 to 127.0.0.1:52103 2023-03-29 03:56:05,164 DEBUG [Thread-3259] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-29 03:56:05,305 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/A/52f9b43ca77f451a9ae0f95a9d2e92e4 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/52f9b43ca77f451a9ae0f95a9d2e92e4 2023-03-29 03:56:05,308 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0337be0f939be2a2b593eefecee32758/A of 0337be0f939be2a2b593eefecee32758 into 52f9b43ca77f451a9ae0f95a9d2e92e4(size=13.6 K), total size for store is 13.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:56:05,309 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0337be0f939be2a2b593eefecee32758: 2023-03-29 03:56:05,309 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758., storeName=0337be0f939be2a2b593eefecee32758/A, priority=13, startTime=1680062164847; duration=0sec 2023-03-29 03:56:05,309 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:56:05,342 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/B/138b4fa921154526b66314217bbfd2eb as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/138b4fa921154526b66314217bbfd2eb 2023-03-29 03:56:05,345 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 0337be0f939be2a2b593eefecee32758/B of 0337be0f939be2a2b593eefecee32758 into 138b4fa921154526b66314217bbfd2eb(size=13.6 K), total size for store is 13.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:56:05,346 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 0337be0f939be2a2b593eefecee32758: 2023-03-29 03:56:05,346 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758., storeName=0337be0f939be2a2b593eefecee32758/B, priority=13, startTime=1680062164847; duration=0sec 2023-03-29 03:56:05,346 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:56:05,354 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=734 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/B/5535a98390a04ccabe1b7d3b0cfb2863 2023-03-29 03:56:05,764 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=734 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/C/03b3d6d3c9a645f4a323b814ae3cca9f 2023-03-29 03:56:05,767 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/A/9aced92a98c64f1f96abcde50aa2b7c9 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/9aced92a98c64f1f96abcde50aa2b7c9 2023-03-29 03:56:05,770 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/9aced92a98c64f1f96abcde50aa2b7c9, entries=150, sequenceid=734, filesize=11.9 K 2023-03-29 03:56:05,770 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/B/5535a98390a04ccabe1b7d3b0cfb2863 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/5535a98390a04ccabe1b7d3b0cfb2863 2023-03-29 03:56:05,773 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/5535a98390a04ccabe1b7d3b0cfb2863, entries=150, sequenceid=734, filesize=11.9 K 2023-03-29 03:56:05,773 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/C/03b3d6d3c9a645f4a323b814ae3cca9f as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/03b3d6d3c9a645f4a323b814ae3cca9f 2023-03-29 03:56:05,776 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/03b3d6d3c9a645f4a323b814ae3cca9f, entries=150, sequenceid=734, filesize=11.9 K 2023-03-29 03:56:05,777 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=26.84 KB/27480 for 0337be0f939be2a2b593eefecee32758 in 897ms, sequenceid=734, compaction requested=false 2023-03-29 03:56:05,777 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 0337be0f939be2a2b593eefecee32758: 2023-03-29 03:56:05,777 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:56:05,777 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:56:05,777 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:56:05,777 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:05,777 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:56:05,777 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:56:05,778 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:05,778 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:05,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:05,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:05,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:05,779 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:05,779 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:56:05,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:05,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:05,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:05,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:05,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:05,780 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:56:05,780 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@d51493a[Count = 0] remaining members to acquire global barrier 2023-03-29 03:56:05,780 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:56:05,780 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:05,781 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:05,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:05,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:05,781 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:56:05,781 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:56:05,781 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:56:05,781 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:05,781 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:56:05,782 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:56:05,782 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:05,783 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:05,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:05,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:05,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:05,783 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:56:05,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:05,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:05,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:05,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:05,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:05,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:05,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:05,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:56:05,784 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:56:05,784 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:56:05,784 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:56:05,784 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:56:05,785 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:56:05,785 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:56:05,787 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:05,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:05,787 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:05,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:05,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:05,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:05,787 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:05,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:05,787 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:05,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:05,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:05,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:05,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:05,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:05,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:05,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:05,788 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:05,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:05,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:05,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:05,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:05,790 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:05,790 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:05,790 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:05,790 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:05,790 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:05,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:05,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:05,790 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:05,790 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:56:05,790 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:05,790 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:56:05,790 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:56:05,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:05,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:05,790 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:05,790 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:05,790 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:05,791 DEBUG [Thread-3261] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:56:05,791 DEBUG [Thread-3261] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:56:05,891 DEBUG [Thread-3261] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:56:05,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:56:14,696 DEBUG [Thread-3251] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5726fb8b to 127.0.0.1:52103 2023-03-29 03:56:14,696 DEBUG [Thread-3251] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-29 03:56:14,696 INFO [Listener at localhost/35027] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-03-29 03:56:14,696 INFO [Listener at localhost/35027] hbase.AcidGuaranteesTestTool(394): wrote 8 2023-03-29 03:56:14,696 INFO [Listener at localhost/35027] hbase.AcidGuaranteesTestTool(394): wrote 99 2023-03-29 03:56:14,697 INFO [Listener at localhost/35027] hbase.AcidGuaranteesTestTool(394): wrote 188 2023-03-29 03:56:14,697 INFO [Listener at localhost/35027] hbase.AcidGuaranteesTestTool(394): wrote 93 2023-03-29 03:56:14,697 INFO [Listener at localhost/35027] hbase.AcidGuaranteesTestTool(394): wrote 194 2023-03-29 03:56:14,697 INFO [Listener at localhost/35027] hbase.AcidGuaranteesTestTool(396): Readers: 2023-03-29 03:56:14,697 INFO [Listener at localhost/35027] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-03-29 03:56:14,697 INFO [Listener at localhost/35027] hbase.AcidGuaranteesTestTool(402): scanned 1868 2023-03-29 03:56:14,697 INFO [Listener at localhost/35027] hbase.AcidGuaranteesTestTool(403): verified 5600 rows 2023-03-29 03:56:14,697 INFO [Listener at localhost/35027] hbase.AcidGuaranteesTestTool(402): scanned 1877 2023-03-29 03:56:14,697 INFO [Listener at localhost/35027] hbase.AcidGuaranteesTestTool(403): verified 5624 rows 2023-03-29 03:56:14,697 INFO [Listener at localhost/35027] hbase.AcidGuaranteesTestTool(402): scanned 1868 2023-03-29 03:56:14,697 INFO [Listener at localhost/35027] hbase.AcidGuaranteesTestTool(403): verified 5604 rows 2023-03-29 03:56:14,697 INFO [Listener at localhost/35027] hbase.AcidGuaranteesTestTool(402): scanned 1864 2023-03-29 03:56:14,697 INFO [Listener at localhost/35027] hbase.AcidGuaranteesTestTool(403): verified 5591 rows 2023-03-29 03:56:14,697 INFO [Listener at localhost/35027] hbase.AcidGuaranteesTestTool(402): scanned 1874 2023-03-29 03:56:14,697 INFO [Listener at localhost/35027] hbase.AcidGuaranteesTestTool(403): verified 5620 rows 2023-03-29 03:56:14,697 INFO [Listener at localhost/35027] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-03-29 03:56:14,697 DEBUG [Listener at localhost/35027] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x34ad4aa8 to 127.0.0.1:52103 2023-03-29 03:56:14,697 DEBUG [Listener at localhost/35027] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-29 03:56:14,699 INFO [Listener at localhost/35027] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-03-29 03:56:14,699 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-03-29 03:56:14,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure2.ProcedureExecutor(1029): Stored pid=50, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-03-29 03:56:14,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-03-29 03:56:14,704 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680062174704"}]},"ts":"1680062174704"} 2023-03-29 03:56:14,705 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-03-29 03:56:14,706 INFO [PEWorker-2] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-03-29 03:56:14,707 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=51, ppid=50, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=0337be0f939be2a2b593eefecee32758, UNASSIGN}] 2023-03-29 03:56:14,708 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=51, ppid=50, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=0337be0f939be2a2b593eefecee32758, UNASSIGN 2023-03-29 03:56:14,709 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=0337be0f939be2a2b593eefecee32758, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:14,709 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680062174709"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680062174709"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680062174709"}]},"ts":"1680062174709"} 2023-03-29 03:56:14,709 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=52, ppid=51, state=RUNNABLE; CloseRegionProcedure 0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534}] 2023-03-29 03:56:14,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-03-29 03:56:14,861 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 0337be0f939be2a2b593eefecee32758 2023-03-29 03:56:14,861 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 0337be0f939be2a2b593eefecee32758, disabling compactions & flushes 2023-03-29 03:56:14,861 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:56:14,861 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:56:14,861 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. after waiting 0 ms 2023-03-29 03:56:14,861 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:56:14,861 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 0337be0f939be2a2b593eefecee32758 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-03-29 03:56:14,862 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0337be0f939be2a2b593eefecee32758, store=A 2023-03-29 03:56:14,862 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:56:14,862 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0337be0f939be2a2b593eefecee32758, store=B 2023-03-29 03:56:14,862 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:56:14,862 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 0337be0f939be2a2b593eefecee32758, store=C 2023-03-29 03:56:14,862 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:56:15,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-03-29 03:56:15,269 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=745 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/A/67a09e2d913944b793f866eef4b66e3d 2023-03-29 03:56:15,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-03-29 03:56:15,678 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=745 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/B/03de53776b84453d81f139bcdfc94e4f 2023-03-29 03:56:15,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-03-29 03:56:16,087 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=745 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/C/32101820ce3e4c42b6e9fad1b0c5bd55 2023-03-29 03:56:16,090 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/A/67a09e2d913944b793f866eef4b66e3d as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/67a09e2d913944b793f866eef4b66e3d 2023-03-29 03:56:16,093 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/67a09e2d913944b793f866eef4b66e3d, entries=150, sequenceid=745, filesize=11.9 K 2023-03-29 03:56:16,094 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/B/03de53776b84453d81f139bcdfc94e4f as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/03de53776b84453d81f139bcdfc94e4f 2023-03-29 03:56:16,097 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/03de53776b84453d81f139bcdfc94e4f, entries=150, sequenceid=745, filesize=11.9 K 2023-03-29 03:56:16,097 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/.tmp/C/32101820ce3e4c42b6e9fad1b0c5bd55 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/32101820ce3e4c42b6e9fad1b0c5bd55 2023-03-29 03:56:16,100 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/32101820ce3e4c42b6e9fad1b0c5bd55, entries=150, sequenceid=745, filesize=11.9 K 2023-03-29 03:56:16,100 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=0 B/0 for 0337be0f939be2a2b593eefecee32758 in 1239ms, sequenceid=745, compaction requested=true 2023-03-29 03:56:16,102 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/864bc06b60e74a85addefd04b73687ac, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/1e00e4ccd2934f819164fee5c6951415, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/870bab9e58f14816ac5fa8dd8f3319c3, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/5cf050c8a8df41719a04e46bc3dcc78e, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/6e49ac1f32944776aad64cae500aa7b5, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/f7621e1159ef42bb8ebb6ff0f78b57d3, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/1416602e3e27463ab58f12fe33aeed04, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/400588da025049dbaaeb6f7e5d267f09, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/4b7b48c3c92f4326a91c49a708364441, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/557ebd52497442e89b55a5ef6402788e, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/6208b63e83a2467fab77a17ffd027841, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/d7a37ee0bd1d46ebbeb7d45610d8bf92, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/462210eb463f48a793a5da760b1f06c4, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/7b242d42d4e6411b9bb939897c7fb883, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/d1ae589112414c0094e6c2ec76a39979, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/5e309c756055447bb1b9526249ab5159, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/e1129ad98da248ba8a9cc4aca7797dc0, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/fbb43625dbe84b1c9c268879845860b8, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/0a726acce5094c06a95800bd6276db19, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/68f036894ad443fdb6aa41eb6bd4b127, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/9cb1db4d28db4c4884d5c646b4b2be8c, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/0cc29faa019e4fef853c61dd1783e531, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/46961c48b11f4222a0a52f50c284ed12, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/d3b472e3f9754511a3c555392a7286a0, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/8ab5930905054ddbb64101af67bdaa9c, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/3927b7ab768b4f0fbfc2cef0f955a8cd, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/dfe69b2dd25d4dd0b91387ae0e2d41e4, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/28762f05a76f4389ae4e2b94d00c18d9, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/4db6ee64cb5a4af4a1e13f51ebe3176f, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/7141a92d45374ef8920636399cf79454, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/51f8501cb4754894a9f1e65d7ddfafd1, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/f42a3d7080724d9f990f9d27dc19e9ac, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/c0f72296859d4937be1da21f5d2ce54d, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/d4165a46320a4f998fe47c5dc08a2bd0, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/613125bcca8045bdbab5d5850fdc9036, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/52bc362d09cd4b8a834ff56e0ba0a863, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/7f05f3f8671042b49bca3bed9e75b114, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/6ea17ae149da4fc8914a8e7b2455fe92, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/174d05012b6c42c09dea0b6d8aff6366, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/c345e9e534f442c094c75bbd109fbeaa, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/8ddf6e37a8d7407cafe2ff2b319c2c93, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/cfbc79711a644e11975b0b79e7bfc352, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/d34ed3d3a6ca43ca95085bfd5f40febf, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/5d72012d115b4fa094ec80ef59eed0eb, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/a8a62a6bd82245178e88df4cd7b26ad1, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/655cd655a2714e678f9ce1310ef465f2, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/50d00105bf4848c1b04809f888c36e56, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/66a52fd08fa440ea945ccef1c996eee1, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/f658f3ea5f51467d83cc627d02bdc291, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/b009847b456c439387422ef49d1987e1, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/42a66d4e8a1347199c27b67c55cfa960] to archive 2023-03-29 03:56:16,104 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-29 03:56:16,105 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/864bc06b60e74a85addefd04b73687ac to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/864bc06b60e74a85addefd04b73687ac 2023-03-29 03:56:16,107 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/1e00e4ccd2934f819164fee5c6951415 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/1e00e4ccd2934f819164fee5c6951415 2023-03-29 03:56:16,108 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/870bab9e58f14816ac5fa8dd8f3319c3 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/870bab9e58f14816ac5fa8dd8f3319c3 2023-03-29 03:56:16,109 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/5cf050c8a8df41719a04e46bc3dcc78e to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/5cf050c8a8df41719a04e46bc3dcc78e 2023-03-29 03:56:16,110 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/6e49ac1f32944776aad64cae500aa7b5 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/6e49ac1f32944776aad64cae500aa7b5 2023-03-29 03:56:16,111 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/f7621e1159ef42bb8ebb6ff0f78b57d3 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/f7621e1159ef42bb8ebb6ff0f78b57d3 2023-03-29 03:56:16,112 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/1416602e3e27463ab58f12fe33aeed04 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/1416602e3e27463ab58f12fe33aeed04 2023-03-29 03:56:16,113 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/400588da025049dbaaeb6f7e5d267f09 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/400588da025049dbaaeb6f7e5d267f09 2023-03-29 03:56:16,113 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/4b7b48c3c92f4326a91c49a708364441 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/4b7b48c3c92f4326a91c49a708364441 2023-03-29 03:56:16,114 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/557ebd52497442e89b55a5ef6402788e to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/557ebd52497442e89b55a5ef6402788e 2023-03-29 03:56:16,115 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/6208b63e83a2467fab77a17ffd027841 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/6208b63e83a2467fab77a17ffd027841 2023-03-29 03:56:16,116 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/d7a37ee0bd1d46ebbeb7d45610d8bf92 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/d7a37ee0bd1d46ebbeb7d45610d8bf92 2023-03-29 03:56:16,117 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/462210eb463f48a793a5da760b1f06c4 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/462210eb463f48a793a5da760b1f06c4 2023-03-29 03:56:16,118 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/7b242d42d4e6411b9bb939897c7fb883 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/7b242d42d4e6411b9bb939897c7fb883 2023-03-29 03:56:16,119 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/d1ae589112414c0094e6c2ec76a39979 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/d1ae589112414c0094e6c2ec76a39979 2023-03-29 03:56:16,119 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/5e309c756055447bb1b9526249ab5159 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/5e309c756055447bb1b9526249ab5159 2023-03-29 03:56:16,120 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/e1129ad98da248ba8a9cc4aca7797dc0 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/e1129ad98da248ba8a9cc4aca7797dc0 2023-03-29 03:56:16,121 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/fbb43625dbe84b1c9c268879845860b8 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/fbb43625dbe84b1c9c268879845860b8 2023-03-29 03:56:16,122 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/0a726acce5094c06a95800bd6276db19 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/0a726acce5094c06a95800bd6276db19 2023-03-29 03:56:16,123 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/68f036894ad443fdb6aa41eb6bd4b127 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/68f036894ad443fdb6aa41eb6bd4b127 2023-03-29 03:56:16,124 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/9cb1db4d28db4c4884d5c646b4b2be8c to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/9cb1db4d28db4c4884d5c646b4b2be8c 2023-03-29 03:56:16,125 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/0cc29faa019e4fef853c61dd1783e531 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/0cc29faa019e4fef853c61dd1783e531 2023-03-29 03:56:16,125 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/46961c48b11f4222a0a52f50c284ed12 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/46961c48b11f4222a0a52f50c284ed12 2023-03-29 03:56:16,126 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/d3b472e3f9754511a3c555392a7286a0 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/d3b472e3f9754511a3c555392a7286a0 2023-03-29 03:56:16,127 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/8ab5930905054ddbb64101af67bdaa9c to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/8ab5930905054ddbb64101af67bdaa9c 2023-03-29 03:56:16,128 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/3927b7ab768b4f0fbfc2cef0f955a8cd to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/3927b7ab768b4f0fbfc2cef0f955a8cd 2023-03-29 03:56:16,129 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/dfe69b2dd25d4dd0b91387ae0e2d41e4 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/dfe69b2dd25d4dd0b91387ae0e2d41e4 2023-03-29 03:56:16,130 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/28762f05a76f4389ae4e2b94d00c18d9 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/28762f05a76f4389ae4e2b94d00c18d9 2023-03-29 03:56:16,130 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/4db6ee64cb5a4af4a1e13f51ebe3176f to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/4db6ee64cb5a4af4a1e13f51ebe3176f 2023-03-29 03:56:16,131 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/7141a92d45374ef8920636399cf79454 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/7141a92d45374ef8920636399cf79454 2023-03-29 03:56:16,132 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/51f8501cb4754894a9f1e65d7ddfafd1 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/51f8501cb4754894a9f1e65d7ddfafd1 2023-03-29 03:56:16,133 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/f42a3d7080724d9f990f9d27dc19e9ac to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/f42a3d7080724d9f990f9d27dc19e9ac 2023-03-29 03:56:16,134 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/c0f72296859d4937be1da21f5d2ce54d to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/c0f72296859d4937be1da21f5d2ce54d 2023-03-29 03:56:16,135 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/d4165a46320a4f998fe47c5dc08a2bd0 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/d4165a46320a4f998fe47c5dc08a2bd0 2023-03-29 03:56:16,135 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/613125bcca8045bdbab5d5850fdc9036 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/613125bcca8045bdbab5d5850fdc9036 2023-03-29 03:56:16,136 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/52bc362d09cd4b8a834ff56e0ba0a863 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/52bc362d09cd4b8a834ff56e0ba0a863 2023-03-29 03:56:16,137 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/7f05f3f8671042b49bca3bed9e75b114 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/7f05f3f8671042b49bca3bed9e75b114 2023-03-29 03:56:16,138 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/6ea17ae149da4fc8914a8e7b2455fe92 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/6ea17ae149da4fc8914a8e7b2455fe92 2023-03-29 03:56:16,139 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/174d05012b6c42c09dea0b6d8aff6366 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/174d05012b6c42c09dea0b6d8aff6366 2023-03-29 03:56:16,139 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/c345e9e534f442c094c75bbd109fbeaa to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/c345e9e534f442c094c75bbd109fbeaa 2023-03-29 03:56:16,140 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/8ddf6e37a8d7407cafe2ff2b319c2c93 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/8ddf6e37a8d7407cafe2ff2b319c2c93 2023-03-29 03:56:16,141 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/cfbc79711a644e11975b0b79e7bfc352 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/cfbc79711a644e11975b0b79e7bfc352 2023-03-29 03:56:16,142 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/d34ed3d3a6ca43ca95085bfd5f40febf to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/d34ed3d3a6ca43ca95085bfd5f40febf 2023-03-29 03:56:16,142 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/5d72012d115b4fa094ec80ef59eed0eb to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/5d72012d115b4fa094ec80ef59eed0eb 2023-03-29 03:56:16,143 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/a8a62a6bd82245178e88df4cd7b26ad1 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/a8a62a6bd82245178e88df4cd7b26ad1 2023-03-29 03:56:16,144 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/655cd655a2714e678f9ce1310ef465f2 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/655cd655a2714e678f9ce1310ef465f2 2023-03-29 03:56:16,145 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/50d00105bf4848c1b04809f888c36e56 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/50d00105bf4848c1b04809f888c36e56 2023-03-29 03:56:16,146 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/66a52fd08fa440ea945ccef1c996eee1 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/66a52fd08fa440ea945ccef1c996eee1 2023-03-29 03:56:16,146 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/f658f3ea5f51467d83cc627d02bdc291 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/f658f3ea5f51467d83cc627d02bdc291 2023-03-29 03:56:16,147 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/b009847b456c439387422ef49d1987e1 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/b009847b456c439387422ef49d1987e1 2023-03-29 03:56:16,148 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/42a66d4e8a1347199c27b67c55cfa960 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/42a66d4e8a1347199c27b67c55cfa960 2023-03-29 03:56:16,151 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/2cda48bbe6884a9ebddda4f078f5687d, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/2614cf7ffcc743e3bdd7fbc3310a4690, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/810d71e247684531a42f751bff5c4531, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/7ba395cadaaa42cfb3593a08fb446132, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/4da76b6e515a45128573ff35877ed44b, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/ff38438e894640a9a40125a321c52993, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/158c8b947a6245478838e4719cbe8341, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/f77132a4406e418bb8ea6dc33ec073d1, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/0233060e9cbe4c188069a55788552a3e, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/29ee69f7d34248348ca914a13a0b6928, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/9d4f50edeef7455cb4751971c6177661, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/29eb1917b4d841779e0f57359d1101f0, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/19a197899dd3467b9f5599e1c6d707bc, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/5b0d803ed6f54678bb227f66581a9faf, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/ad4dd405c3714039abc133505b1c57dc, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/c288bc52f45740fba3a967ed962ba6a1, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/ea3489642ac446068bb0ec3a0b7dac96, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/02cb896a46b44d96862d932e28c317bd, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/bfabbcc60fdf4b76bf1701a2dd4b285d, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/585023d84be347cfae77d25f72e093a5, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/02125a7e4e2844d59f622f66e4a647ed, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/0167580ea68e417dbca64eec84b86414, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/6460c1d108414bbfab636286877c294e, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/4a490c12a32b450b89b4fc545245cad8, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/a1dd3e80578145538dd856018625ba93, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/a864856115004230843d2ec088e64dea, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/844eafdd81e846a7a821b9853a7a670c, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/623e374564df4490ab343b2bedecc9ef, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/034bb61da4d740b28952a27021b0526c, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/40f05e34f4cf4856930507d82699303b, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/108861cc9bf849a58c9072eabc5962de, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/534d4694ad7d438397e2812bc07d748e, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/90d4c81ecf2346cc90aab26a958a6d0b, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/39c4248edeb44cf889f63c1f1e3f3417, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/31672a1c8ab14c97a2a0c892081d42f0, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/04e6e48575f4443887aac3a6f3d7d176, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/668d41bbbf754f38883ac62f93892179, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/9ad52d585ccd4d32b332da3e0a92a056, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/4c89102314124fe6873b294cbb23270e, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/623594276a2e45c2be7f2fde45197a17, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/989d7e0a0235425e80c6cf199cf2fd91, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/63face3b15d34951ab280bce5d0248c8, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/e0c058ab07614f989f11027815697f91, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/13a6ca1c587a49a2b16109f0e4e8a872, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/e2ad3864577a4fd3ae618133e5e9adc1, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/4088eba980fd4881981cf6c282370dc5, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/8ec7a596d0994c529676ff038ba7fe84, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/2ee6ec5e22064976bb14b0af3a4f5631, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/957a9a99004145f6af5d3147ad15f88e, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/f2a825f4ff7b484088f9f817dee7c8f0, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/2aa398b21223491a89876fa170a34762] to archive 2023-03-29 03:56:16,152 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-29 03:56:16,153 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/2cda48bbe6884a9ebddda4f078f5687d to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/2cda48bbe6884a9ebddda4f078f5687d 2023-03-29 03:56:16,154 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/2614cf7ffcc743e3bdd7fbc3310a4690 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/2614cf7ffcc743e3bdd7fbc3310a4690 2023-03-29 03:56:16,155 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/810d71e247684531a42f751bff5c4531 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/810d71e247684531a42f751bff5c4531 2023-03-29 03:56:16,155 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/7ba395cadaaa42cfb3593a08fb446132 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/7ba395cadaaa42cfb3593a08fb446132 2023-03-29 03:56:16,156 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/4da76b6e515a45128573ff35877ed44b to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/4da76b6e515a45128573ff35877ed44b 2023-03-29 03:56:16,157 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/ff38438e894640a9a40125a321c52993 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/ff38438e894640a9a40125a321c52993 2023-03-29 03:56:16,158 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/158c8b947a6245478838e4719cbe8341 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/158c8b947a6245478838e4719cbe8341 2023-03-29 03:56:16,158 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/f77132a4406e418bb8ea6dc33ec073d1 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/f77132a4406e418bb8ea6dc33ec073d1 2023-03-29 03:56:16,159 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/0233060e9cbe4c188069a55788552a3e to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/0233060e9cbe4c188069a55788552a3e 2023-03-29 03:56:16,160 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/29ee69f7d34248348ca914a13a0b6928 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/29ee69f7d34248348ca914a13a0b6928 2023-03-29 03:56:16,161 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/9d4f50edeef7455cb4751971c6177661 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/9d4f50edeef7455cb4751971c6177661 2023-03-29 03:56:16,161 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/29eb1917b4d841779e0f57359d1101f0 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/29eb1917b4d841779e0f57359d1101f0 2023-03-29 03:56:16,162 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/19a197899dd3467b9f5599e1c6d707bc to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/19a197899dd3467b9f5599e1c6d707bc 2023-03-29 03:56:16,163 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/5b0d803ed6f54678bb227f66581a9faf to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/5b0d803ed6f54678bb227f66581a9faf 2023-03-29 03:56:16,173 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/ad4dd405c3714039abc133505b1c57dc to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/ad4dd405c3714039abc133505b1c57dc 2023-03-29 03:56:16,174 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/c288bc52f45740fba3a967ed962ba6a1 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/c288bc52f45740fba3a967ed962ba6a1 2023-03-29 03:56:16,175 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/ea3489642ac446068bb0ec3a0b7dac96 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/ea3489642ac446068bb0ec3a0b7dac96 2023-03-29 03:56:16,176 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/02cb896a46b44d96862d932e28c317bd to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/02cb896a46b44d96862d932e28c317bd 2023-03-29 03:56:16,176 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/bfabbcc60fdf4b76bf1701a2dd4b285d to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/bfabbcc60fdf4b76bf1701a2dd4b285d 2023-03-29 03:56:16,177 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/585023d84be347cfae77d25f72e093a5 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/585023d84be347cfae77d25f72e093a5 2023-03-29 03:56:16,178 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/02125a7e4e2844d59f622f66e4a647ed to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/02125a7e4e2844d59f622f66e4a647ed 2023-03-29 03:56:16,179 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/0167580ea68e417dbca64eec84b86414 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/0167580ea68e417dbca64eec84b86414 2023-03-29 03:56:16,180 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/6460c1d108414bbfab636286877c294e to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/6460c1d108414bbfab636286877c294e 2023-03-29 03:56:16,181 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/4a490c12a32b450b89b4fc545245cad8 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/4a490c12a32b450b89b4fc545245cad8 2023-03-29 03:56:16,181 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/a1dd3e80578145538dd856018625ba93 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/a1dd3e80578145538dd856018625ba93 2023-03-29 03:56:16,182 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/a864856115004230843d2ec088e64dea to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/a864856115004230843d2ec088e64dea 2023-03-29 03:56:16,183 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/844eafdd81e846a7a821b9853a7a670c to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/844eafdd81e846a7a821b9853a7a670c 2023-03-29 03:56:16,184 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/623e374564df4490ab343b2bedecc9ef to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/623e374564df4490ab343b2bedecc9ef 2023-03-29 03:56:16,185 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/034bb61da4d740b28952a27021b0526c to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/034bb61da4d740b28952a27021b0526c 2023-03-29 03:56:16,186 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/40f05e34f4cf4856930507d82699303b to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/40f05e34f4cf4856930507d82699303b 2023-03-29 03:56:16,186 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/108861cc9bf849a58c9072eabc5962de to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/108861cc9bf849a58c9072eabc5962de 2023-03-29 03:56:16,187 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/534d4694ad7d438397e2812bc07d748e to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/534d4694ad7d438397e2812bc07d748e 2023-03-29 03:56:16,188 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/90d4c81ecf2346cc90aab26a958a6d0b to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/90d4c81ecf2346cc90aab26a958a6d0b 2023-03-29 03:56:16,189 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/39c4248edeb44cf889f63c1f1e3f3417 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/39c4248edeb44cf889f63c1f1e3f3417 2023-03-29 03:56:16,190 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/31672a1c8ab14c97a2a0c892081d42f0 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/31672a1c8ab14c97a2a0c892081d42f0 2023-03-29 03:56:16,191 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/04e6e48575f4443887aac3a6f3d7d176 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/04e6e48575f4443887aac3a6f3d7d176 2023-03-29 03:56:16,192 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/668d41bbbf754f38883ac62f93892179 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/668d41bbbf754f38883ac62f93892179 2023-03-29 03:56:16,193 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/9ad52d585ccd4d32b332da3e0a92a056 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/9ad52d585ccd4d32b332da3e0a92a056 2023-03-29 03:56:16,194 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/4c89102314124fe6873b294cbb23270e to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/4c89102314124fe6873b294cbb23270e 2023-03-29 03:56:16,195 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/623594276a2e45c2be7f2fde45197a17 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/623594276a2e45c2be7f2fde45197a17 2023-03-29 03:56:16,196 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/989d7e0a0235425e80c6cf199cf2fd91 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/989d7e0a0235425e80c6cf199cf2fd91 2023-03-29 03:56:16,197 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/63face3b15d34951ab280bce5d0248c8 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/63face3b15d34951ab280bce5d0248c8 2023-03-29 03:56:16,198 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/e0c058ab07614f989f11027815697f91 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/e0c058ab07614f989f11027815697f91 2023-03-29 03:56:16,199 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/13a6ca1c587a49a2b16109f0e4e8a872 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/13a6ca1c587a49a2b16109f0e4e8a872 2023-03-29 03:56:16,200 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/e2ad3864577a4fd3ae618133e5e9adc1 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/e2ad3864577a4fd3ae618133e5e9adc1 2023-03-29 03:56:16,201 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/4088eba980fd4881981cf6c282370dc5 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/4088eba980fd4881981cf6c282370dc5 2023-03-29 03:56:16,202 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/8ec7a596d0994c529676ff038ba7fe84 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/8ec7a596d0994c529676ff038ba7fe84 2023-03-29 03:56:16,203 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/2ee6ec5e22064976bb14b0af3a4f5631 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/2ee6ec5e22064976bb14b0af3a4f5631 2023-03-29 03:56:16,204 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/957a9a99004145f6af5d3147ad15f88e to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/957a9a99004145f6af5d3147ad15f88e 2023-03-29 03:56:16,205 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/f2a825f4ff7b484088f9f817dee7c8f0 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/f2a825f4ff7b484088f9f817dee7c8f0 2023-03-29 03:56:16,206 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/2aa398b21223491a89876fa170a34762 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/2aa398b21223491a89876fa170a34762 2023-03-29 03:56:16,209 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/ff07ce7ad8024f97a6be2f47b92a3b03, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/ec12ce726f5e4f96a8d5920ca5912b90, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/61f489be14cd44eda718675e5f141996, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/3a317a92178b45a99a492e78a5ff7e2b, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/2847721ff5964bb18fb5aba244c7f802, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/9a96ecf7a8134fce99f2d5c33f57647f, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/7854b5d0a7bd4202ba02a1fc8b3dd62a, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/c29e41d57b164baa98de1a169f6a6597, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/0fad591a5f4c4b3092427eb05ba2a9df, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/82d6e07a2f4b4661a7dfd3be6f7bc65b, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/a515905a06bd485e9634b6af7713c07b, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/bf79062a472342099e8940b53d90c888, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/23bbbea184ec4585943bac033c88f5ac, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/fb54b30d187644b9abcd14be0da33059, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/dcf0e1b3ab894c2b9078c5157207ebf3, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/c7d9d065f5924811bbd65187b9c47eec, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/30b14896f056491e8afcc3c6d49197ab, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/e18a0008c2ed428babfa74b8c2f411e9, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/eb4a7aafeb7b4e068c8b7db465c15f54, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/097531d3a75941288a3e28d3ff19e344, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/5b69893bd5314bc88f715dbae7065fae, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/c47493d2a47542af926eb7208eb11c00, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/f1477162972d4ec9b21f5e9f23acf144, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/b86beaf8145a40dc9dac63307bcdf306, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/ea547bd02fc34e77aa006ef48572d607, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/0ee4e9ae9a1140c4a619e0d04219eee3, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/ae7f434d973b4520819f2fed2e18375b, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/89e15f533c40459883faf24b666e822c, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/b2d96e8382364215ba9580656d165323, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/015c0e30e7ef4d1b9f79f32b1abca9f5, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/cf0b02467c05415381ca134a3e7a61e8, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/18afc47121e34a40bf5e67294505ff48, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/060c4b6e41544963b446beb0ef6415e8, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/d6eba1304e794028a00feae7ceece564, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/697b3bcc193d48b1af4772e35c465734, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/a576371842fc41e889197e2cd8db3f4a, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/1136fa759b5e46a38ba3804b086bbe5b, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/43d18a2585fb40c4bf0ac76ef9222924, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/1b0a9b5703d44107868249c7c14ea58a, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/e6bdd06d64a7451d9f26308f0e362f63, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/109e3df5aa334d31b94d54b75a7f09cf, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/bab00af85694494bbf8b1c24a2e43df7, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/3db623b7a43d4f1c85f6275ca14b09b7, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/564227d153394044a02da50f51902990, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/d28603b66bb7487a89adf3dd518c380d, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/90677e947d784b5ab6b3c7f7a29f06a5, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/2b65898d1a6b4249a1087f8fea67a104, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/8083af365b3049a6b5b315ee7139b77d, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/d720438a74b843edb3f074a46809bcdb, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/184a069a2097447d8fb16d1485190ddd] to archive 2023-03-29 03:56:16,210 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-29 03:56:16,211 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/ff07ce7ad8024f97a6be2f47b92a3b03 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/ff07ce7ad8024f97a6be2f47b92a3b03 2023-03-29 03:56:16,212 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/ec12ce726f5e4f96a8d5920ca5912b90 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/ec12ce726f5e4f96a8d5920ca5912b90 2023-03-29 03:56:16,213 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/61f489be14cd44eda718675e5f141996 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/61f489be14cd44eda718675e5f141996 2023-03-29 03:56:16,214 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/3a317a92178b45a99a492e78a5ff7e2b to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/3a317a92178b45a99a492e78a5ff7e2b 2023-03-29 03:56:16,215 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/2847721ff5964bb18fb5aba244c7f802 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/2847721ff5964bb18fb5aba244c7f802 2023-03-29 03:56:16,216 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/9a96ecf7a8134fce99f2d5c33f57647f to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/9a96ecf7a8134fce99f2d5c33f57647f 2023-03-29 03:56:16,217 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/7854b5d0a7bd4202ba02a1fc8b3dd62a to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/7854b5d0a7bd4202ba02a1fc8b3dd62a 2023-03-29 03:56:16,218 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/c29e41d57b164baa98de1a169f6a6597 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/c29e41d57b164baa98de1a169f6a6597 2023-03-29 03:56:16,219 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/0fad591a5f4c4b3092427eb05ba2a9df to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/0fad591a5f4c4b3092427eb05ba2a9df 2023-03-29 03:56:16,220 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/82d6e07a2f4b4661a7dfd3be6f7bc65b to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/82d6e07a2f4b4661a7dfd3be6f7bc65b 2023-03-29 03:56:16,221 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/a515905a06bd485e9634b6af7713c07b to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/a515905a06bd485e9634b6af7713c07b 2023-03-29 03:56:16,222 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/bf79062a472342099e8940b53d90c888 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/bf79062a472342099e8940b53d90c888 2023-03-29 03:56:16,223 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/23bbbea184ec4585943bac033c88f5ac to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/23bbbea184ec4585943bac033c88f5ac 2023-03-29 03:56:16,223 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/fb54b30d187644b9abcd14be0da33059 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/fb54b30d187644b9abcd14be0da33059 2023-03-29 03:56:16,224 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/dcf0e1b3ab894c2b9078c5157207ebf3 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/dcf0e1b3ab894c2b9078c5157207ebf3 2023-03-29 03:56:16,225 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/c7d9d065f5924811bbd65187b9c47eec to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/c7d9d065f5924811bbd65187b9c47eec 2023-03-29 03:56:16,226 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/30b14896f056491e8afcc3c6d49197ab to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/30b14896f056491e8afcc3c6d49197ab 2023-03-29 03:56:16,227 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/e18a0008c2ed428babfa74b8c2f411e9 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/e18a0008c2ed428babfa74b8c2f411e9 2023-03-29 03:56:16,228 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/eb4a7aafeb7b4e068c8b7db465c15f54 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/eb4a7aafeb7b4e068c8b7db465c15f54 2023-03-29 03:56:16,228 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/097531d3a75941288a3e28d3ff19e344 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/097531d3a75941288a3e28d3ff19e344 2023-03-29 03:56:16,229 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/5b69893bd5314bc88f715dbae7065fae to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/5b69893bd5314bc88f715dbae7065fae 2023-03-29 03:56:16,230 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/c47493d2a47542af926eb7208eb11c00 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/c47493d2a47542af926eb7208eb11c00 2023-03-29 03:56:16,231 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/f1477162972d4ec9b21f5e9f23acf144 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/f1477162972d4ec9b21f5e9f23acf144 2023-03-29 03:56:16,232 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/b86beaf8145a40dc9dac63307bcdf306 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/b86beaf8145a40dc9dac63307bcdf306 2023-03-29 03:56:16,233 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/ea547bd02fc34e77aa006ef48572d607 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/ea547bd02fc34e77aa006ef48572d607 2023-03-29 03:56:16,234 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/0ee4e9ae9a1140c4a619e0d04219eee3 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/0ee4e9ae9a1140c4a619e0d04219eee3 2023-03-29 03:56:16,234 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/ae7f434d973b4520819f2fed2e18375b to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/ae7f434d973b4520819f2fed2e18375b 2023-03-29 03:56:16,235 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/89e15f533c40459883faf24b666e822c to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/89e15f533c40459883faf24b666e822c 2023-03-29 03:56:16,236 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/b2d96e8382364215ba9580656d165323 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/b2d96e8382364215ba9580656d165323 2023-03-29 03:56:16,237 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/015c0e30e7ef4d1b9f79f32b1abca9f5 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/015c0e30e7ef4d1b9f79f32b1abca9f5 2023-03-29 03:56:16,237 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/cf0b02467c05415381ca134a3e7a61e8 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/cf0b02467c05415381ca134a3e7a61e8 2023-03-29 03:56:16,238 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/18afc47121e34a40bf5e67294505ff48 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/18afc47121e34a40bf5e67294505ff48 2023-03-29 03:56:16,239 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/060c4b6e41544963b446beb0ef6415e8 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/060c4b6e41544963b446beb0ef6415e8 2023-03-29 03:56:16,240 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/d6eba1304e794028a00feae7ceece564 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/d6eba1304e794028a00feae7ceece564 2023-03-29 03:56:16,241 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/697b3bcc193d48b1af4772e35c465734 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/697b3bcc193d48b1af4772e35c465734 2023-03-29 03:56:16,241 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/a576371842fc41e889197e2cd8db3f4a to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/a576371842fc41e889197e2cd8db3f4a 2023-03-29 03:56:16,242 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/1136fa759b5e46a38ba3804b086bbe5b to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/1136fa759b5e46a38ba3804b086bbe5b 2023-03-29 03:56:16,243 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/43d18a2585fb40c4bf0ac76ef9222924 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/43d18a2585fb40c4bf0ac76ef9222924 2023-03-29 03:56:16,244 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/1b0a9b5703d44107868249c7c14ea58a to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/1b0a9b5703d44107868249c7c14ea58a 2023-03-29 03:56:16,245 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/e6bdd06d64a7451d9f26308f0e362f63 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/e6bdd06d64a7451d9f26308f0e362f63 2023-03-29 03:56:16,246 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/109e3df5aa334d31b94d54b75a7f09cf to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/109e3df5aa334d31b94d54b75a7f09cf 2023-03-29 03:56:16,247 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/bab00af85694494bbf8b1c24a2e43df7 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/bab00af85694494bbf8b1c24a2e43df7 2023-03-29 03:56:16,248 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/3db623b7a43d4f1c85f6275ca14b09b7 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/3db623b7a43d4f1c85f6275ca14b09b7 2023-03-29 03:56:16,248 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/564227d153394044a02da50f51902990 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/564227d153394044a02da50f51902990 2023-03-29 03:56:16,249 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/d28603b66bb7487a89adf3dd518c380d to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/d28603b66bb7487a89adf3dd518c380d 2023-03-29 03:56:16,250 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/90677e947d784b5ab6b3c7f7a29f06a5 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/90677e947d784b5ab6b3c7f7a29f06a5 2023-03-29 03:56:16,251 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/2b65898d1a6b4249a1087f8fea67a104 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/2b65898d1a6b4249a1087f8fea67a104 2023-03-29 03:56:16,251 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/8083af365b3049a6b5b315ee7139b77d to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/8083af365b3049a6b5b315ee7139b77d 2023-03-29 03:56:16,252 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/d720438a74b843edb3f074a46809bcdb to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/d720438a74b843edb3f074a46809bcdb 2023-03-29 03:56:16,253 DEBUG [StoreCloser-TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/184a069a2097447d8fb16d1485190ddd to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/184a069a2097447d8fb16d1485190ddd 2023-03-29 03:56:16,257 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/recovered.edits/748.seqid, newMaxSeqId=748, maxSeqId=1 2023-03-29 03:56:16,258 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758. 2023-03-29 03:56:16,258 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 0337be0f939be2a2b593eefecee32758: 2023-03-29 03:56:16,260 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 0337be0f939be2a2b593eefecee32758 2023-03-29 03:56:16,260 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=0337be0f939be2a2b593eefecee32758, regionState=CLOSED 2023-03-29 03:56:16,260 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680062176260"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680062176260"}]},"ts":"1680062176260"} 2023-03-29 03:56:16,262 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=52, resume processing ppid=51 2023-03-29 03:56:16,262 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=52, ppid=51, state=SUCCESS; CloseRegionProcedure 0337be0f939be2a2b593eefecee32758, server=jenkins-hbase4.apache.org,35599,1680062018534 in 1.5520 sec 2023-03-29 03:56:16,263 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=51, resume processing ppid=50 2023-03-29 03:56:16,263 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=51, ppid=50, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=0337be0f939be2a2b593eefecee32758, UNASSIGN in 1.5550 sec 2023-03-29 03:56:16,264 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680062176263"}]},"ts":"1680062176263"} 2023-03-29 03:56:16,264 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-03-29 03:56:16,266 INFO [PEWorker-2] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-03-29 03:56:16,267 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=50, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.5670 sec 2023-03-29 03:56:16,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-03-29 03:56:16,809 INFO [Listener at localhost/35027] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 50 completed 2023-03-29 03:56:16,809 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-03-29 03:56:16,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure2.ProcedureExecutor(1029): Stored pid=53, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-03-29 03:56:16,811 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=53, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-29 03:56:16,812 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=53, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-29 03:56:16,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=53 2023-03-29 03:56:16,816 DEBUG [HFileArchiver-11] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/.tmp/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758 2023-03-29 03:56:16,818 DEBUG [HFileArchiver-11] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/.tmp/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A, FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/.tmp/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B, FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/.tmp/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C, FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/.tmp/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/recovered.edits] 2023-03-29 03:56:16,820 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/.tmp/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/52f9b43ca77f451a9ae0f95a9d2e92e4 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/52f9b43ca77f451a9ae0f95a9d2e92e4 2023-03-29 03:56:16,821 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/.tmp/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/67a09e2d913944b793f866eef4b66e3d to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/67a09e2d913944b793f866eef4b66e3d 2023-03-29 03:56:16,822 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/.tmp/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/9aced92a98c64f1f96abcde50aa2b7c9 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/A/9aced92a98c64f1f96abcde50aa2b7c9 2023-03-29 03:56:16,824 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/.tmp/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/03de53776b84453d81f139bcdfc94e4f to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/03de53776b84453d81f139bcdfc94e4f 2023-03-29 03:56:16,825 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/.tmp/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/138b4fa921154526b66314217bbfd2eb to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/138b4fa921154526b66314217bbfd2eb 2023-03-29 03:56:16,827 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/.tmp/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/5535a98390a04ccabe1b7d3b0cfb2863 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/B/5535a98390a04ccabe1b7d3b0cfb2863 2023-03-29 03:56:16,828 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/.tmp/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/03b3d6d3c9a645f4a323b814ae3cca9f to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/03b3d6d3c9a645f4a323b814ae3cca9f 2023-03-29 03:56:16,829 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/.tmp/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/32101820ce3e4c42b6e9fad1b0c5bd55 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/32101820ce3e4c42b6e9fad1b0c5bd55 2023-03-29 03:56:16,830 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/.tmp/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/67aeeed84f094825ad66c35be23e19a0 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/C/67aeeed84f094825ad66c35be23e19a0 2023-03-29 03:56:16,833 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/.tmp/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/recovered.edits/748.seqid to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758/recovered.edits/748.seqid 2023-03-29 03:56:16,834 DEBUG [HFileArchiver-11] backup.HFileArchiver(596): Deleted hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/.tmp/data/default/TestAcidGuarantees/0337be0f939be2a2b593eefecee32758 2023-03-29 03:56:16,834 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-29 03:56:16,836 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=53, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-29 03:56:16,837 WARN [PEWorker-5] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-03-29 03:56:16,839 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-03-29 03:56:16,839 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=53, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-29 03:56:16,840 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-03-29 03:56:16,840 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1680062176840"}]},"ts":"9223372036854775807"} 2023-03-29 03:56:16,841 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-03-29 03:56:16,841 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 0337be0f939be2a2b593eefecee32758, NAME => 'TestAcidGuarantees,,1680062143950.0337be0f939be2a2b593eefecee32758.', STARTKEY => '', ENDKEY => ''}] 2023-03-29 03:56:16,841 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-03-29 03:56:16,841 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1680062176841"}]},"ts":"9223372036854775807"} 2023-03-29 03:56:16,842 INFO [PEWorker-5] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-03-29 03:56:16,844 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(130): Finished pid=53, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-29 03:56:16,845 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=53, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 35 msec 2023-03-29 03:56:16,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=53 2023-03-29 03:56:16,913 INFO [Listener at localhost/35027] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 53 completed 2023-03-29 03:56:16,927 INFO [Listener at localhost/35027] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testScanAtomicity Thread=225 (was 234), OpenFileDescriptor=567 (was 560) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=392 (was 419), ProcessCount=169 (was 171), AvailableMemoryMB=6984 (was 6594) - AvailableMemoryMB LEAK? - 2023-03-29 03:56:16,938 INFO [Listener at localhost/35027] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobGetAtomicity Thread=225, OpenFileDescriptor=567, MaxFileDescriptor=60000, SystemLoadAverage=392, ProcessCount=169, AvailableMemoryMB=6984 2023-03-29 03:56:16,940 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-29 03:56:16,941 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-29 03:56:16,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure2.ProcedureExecutor(1029): Stored pid=54, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-03-29 03:56:16,943 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-03-29 03:56:16,943 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 54 2023-03-29 03:56:16,943 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-03-29 03:56:16,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-03-29 03:56:16,945 DEBUG [HFileArchiver-12] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/.tmp/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9 2023-03-29 03:56:16,945 DEBUG [HFileArchiver-12] backup.HFileArchiver(153): Directory hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/.tmp/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9 empty. 2023-03-29 03:56:16,945 DEBUG [HFileArchiver-12] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/.tmp/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9 2023-03-29 03:56:16,945 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-29 03:56:17,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-03-29 03:56:17,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-03-29 03:56:17,352 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-29 03:56:17,353 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 94b16ca160750195aa137348cdb80ba9, NAME => 'TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/.tmp 2023-03-29 03:56:17,361 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-29 03:56:17,362 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 94b16ca160750195aa137348cdb80ba9, disabling compactions & flushes 2023-03-29 03:56:17,362 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:17,362 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:17,362 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. after waiting 0 ms 2023-03-29 03:56:17,362 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:17,362 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:17,362 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 94b16ca160750195aa137348cdb80ba9: 2023-03-29 03:56:17,364 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-03-29 03:56:17,365 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680062177365"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680062177365"}]},"ts":"1680062177365"} 2023-03-29 03:56:17,366 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-03-29 03:56:17,366 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-03-29 03:56:17,367 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680062177366"}]},"ts":"1680062177366"} 2023-03-29 03:56:17,367 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-03-29 03:56:17,379 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=94b16ca160750195aa137348cdb80ba9, ASSIGN}] 2023-03-29 03:56:17,381 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=94b16ca160750195aa137348cdb80ba9, ASSIGN 2023-03-29 03:56:17,381 INFO [PEWorker-1] assignment.TransitRegionStateProcedure(193): Starting pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=94b16ca160750195aa137348cdb80ba9, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,35599,1680062018534; forceNewPlan=false, retain=false 2023-03-29 03:56:17,533 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=94b16ca160750195aa137348cdb80ba9, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:17,533 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680062177532"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680062177532"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680062177532"}]},"ts":"1680062177532"} 2023-03-29 03:56:17,535 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=56, ppid=55, state=RUNNABLE; OpenRegionProcedure 94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534}] 2023-03-29 03:56:17,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-03-29 03:56:17,691 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:17,691 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 94b16ca160750195aa137348cdb80ba9, NAME => 'TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.', STARTKEY => '', ENDKEY => ''} 2023-03-29 03:56:17,692 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 94b16ca160750195aa137348cdb80ba9 2023-03-29 03:56:17,692 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-29 03:56:17,692 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 94b16ca160750195aa137348cdb80ba9 2023-03-29 03:56:17,692 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 94b16ca160750195aa137348cdb80ba9 2023-03-29 03:56:17,693 INFO [StoreOpener-94b16ca160750195aa137348cdb80ba9-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 94b16ca160750195aa137348cdb80ba9 2023-03-29 03:56:17,696 DEBUG [StoreOpener-94b16ca160750195aa137348cdb80ba9-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A 2023-03-29 03:56:17,696 DEBUG [StoreOpener-94b16ca160750195aa137348cdb80ba9-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A 2023-03-29 03:56:17,696 INFO [StoreOpener-94b16ca160750195aa137348cdb80ba9-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-29 03:56:17,696 INFO [StoreOpener-94b16ca160750195aa137348cdb80ba9-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 94b16ca160750195aa137348cdb80ba9 columnFamilyName A 2023-03-29 03:56:17,697 INFO [StoreOpener-94b16ca160750195aa137348cdb80ba9-1] regionserver.HStore(310): Store=94b16ca160750195aa137348cdb80ba9/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-29 03:56:17,697 INFO [StoreOpener-94b16ca160750195aa137348cdb80ba9-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 94b16ca160750195aa137348cdb80ba9 2023-03-29 03:56:17,698 DEBUG [StoreOpener-94b16ca160750195aa137348cdb80ba9-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B 2023-03-29 03:56:17,698 DEBUG [StoreOpener-94b16ca160750195aa137348cdb80ba9-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B 2023-03-29 03:56:17,698 INFO [StoreOpener-94b16ca160750195aa137348cdb80ba9-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-29 03:56:17,698 INFO [StoreOpener-94b16ca160750195aa137348cdb80ba9-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 94b16ca160750195aa137348cdb80ba9 columnFamilyName B 2023-03-29 03:56:17,699 INFO [StoreOpener-94b16ca160750195aa137348cdb80ba9-1] regionserver.HStore(310): Store=94b16ca160750195aa137348cdb80ba9/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-29 03:56:17,699 INFO [StoreOpener-94b16ca160750195aa137348cdb80ba9-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 94b16ca160750195aa137348cdb80ba9 2023-03-29 03:56:17,700 DEBUG [StoreOpener-94b16ca160750195aa137348cdb80ba9-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C 2023-03-29 03:56:17,700 DEBUG [StoreOpener-94b16ca160750195aa137348cdb80ba9-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C 2023-03-29 03:56:17,700 INFO [StoreOpener-94b16ca160750195aa137348cdb80ba9-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-29 03:56:17,700 INFO [StoreOpener-94b16ca160750195aa137348cdb80ba9-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 94b16ca160750195aa137348cdb80ba9 columnFamilyName C 2023-03-29 03:56:17,700 INFO [StoreOpener-94b16ca160750195aa137348cdb80ba9-1] regionserver.HStore(310): Store=94b16ca160750195aa137348cdb80ba9/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-29 03:56:17,701 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:17,701 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9 2023-03-29 03:56:17,701 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9 2023-03-29 03:56:17,703 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-29 03:56:17,704 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 94b16ca160750195aa137348cdb80ba9 2023-03-29 03:56:17,706 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-29 03:56:17,706 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 94b16ca160750195aa137348cdb80ba9; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74211954, jitterRate=0.10584428906440735}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-29 03:56:17,706 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 94b16ca160750195aa137348cdb80ba9: 2023-03-29 03:56:17,707 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9., pid=56, masterSystemTime=1680062177686 2023-03-29 03:56:17,708 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:17,708 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:17,708 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=94b16ca160750195aa137348cdb80ba9, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:17,709 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680062177708"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1680062177708"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680062177708"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680062177708"}]},"ts":"1680062177708"} 2023-03-29 03:56:17,710 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=56, resume processing ppid=55 2023-03-29 03:56:17,711 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=56, ppid=55, state=SUCCESS; OpenRegionProcedure 94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 in 175 msec 2023-03-29 03:56:17,712 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=55, resume processing ppid=54 2023-03-29 03:56:17,712 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=55, ppid=54, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=94b16ca160750195aa137348cdb80ba9, ASSIGN in 332 msec 2023-03-29 03:56:17,712 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-03-29 03:56:17,712 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680062177712"}]},"ts":"1680062177712"} 2023-03-29 03:56:17,713 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-03-29 03:56:17,716 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-03-29 03:56:17,717 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=54, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 775 msec 2023-03-29 03:56:18,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-03-29 03:56:18,048 INFO [Listener at localhost/35027] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 54 completed 2023-03-29 03:56:18,049 DEBUG [Listener at localhost/35027] zookeeper.ReadOnlyZKClient(139): Connect 0x59185ba7 to 127.0.0.1:52103 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-29 03:56:18,054 DEBUG [Listener at localhost/35027] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6af74c0e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-29 03:56:18,055 DEBUG [Listener at localhost/35027] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-29 03:56:18,057 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:37636, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-29 03:56:18,057 DEBUG [Listener at localhost/35027] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-03-29 03:56:18,059 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48562, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-03-29 03:56:18,061 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-29 03:56:18,061 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.HMaster$12(2454): Client=jenkins//172.31.14.131 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-29 03:56:18,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure2.ProcedureExecutor(1029): Stored pid=57, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-03-29 03:56:18,475 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-03-29 03:56:18,476 DEBUG [PEWorker-4] util.FSTableDescriptors(519): Deleted hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-29 03:56:18,476 INFO [PEWorker-4] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-03-29 03:56:18,478 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=58, ppid=57, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-03-29 03:56:18,480 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=94b16ca160750195aa137348cdb80ba9, REOPEN/MOVE}] 2023-03-29 03:56:18,481 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=94b16ca160750195aa137348cdb80ba9, REOPEN/MOVE 2023-03-29 03:56:18,481 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=94b16ca160750195aa137348cdb80ba9, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:18,481 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680062178481"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680062178481"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680062178481"}]},"ts":"1680062178481"} 2023-03-29 03:56:18,482 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=60, ppid=59, state=RUNNABLE; CloseRegionProcedure 94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534}] 2023-03-29 03:56:18,634 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 94b16ca160750195aa137348cdb80ba9 2023-03-29 03:56:18,635 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 94b16ca160750195aa137348cdb80ba9, disabling compactions & flushes 2023-03-29 03:56:18,635 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:18,635 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:18,635 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. after waiting 0 ms 2023-03-29 03:56:18,635 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:18,638 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-03-29 03:56:18,638 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:18,638 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 94b16ca160750195aa137348cdb80ba9: 2023-03-29 03:56:18,638 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: 94b16ca160750195aa137348cdb80ba9 to self. 2023-03-29 03:56:18,639 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 94b16ca160750195aa137348cdb80ba9 2023-03-29 03:56:18,640 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=94b16ca160750195aa137348cdb80ba9, regionState=CLOSED 2023-03-29 03:56:18,640 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680062178640"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680062178640"}]},"ts":"1680062178640"} 2023-03-29 03:56:18,642 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=60, resume processing ppid=59 2023-03-29 03:56:18,642 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=60, ppid=59, state=SUCCESS; CloseRegionProcedure 94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 in 159 msec 2023-03-29 03:56:18,642 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=94b16ca160750195aa137348cdb80ba9, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,35599,1680062018534; forceNewPlan=false, retain=true 2023-03-29 03:56:18,793 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=94b16ca160750195aa137348cdb80ba9, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:18,793 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680062178793"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680062178793"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680062178793"}]},"ts":"1680062178793"} 2023-03-29 03:56:18,794 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=61, ppid=59, state=RUNNABLE; OpenRegionProcedure 94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534}] 2023-03-29 03:56:18,948 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:18,949 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 94b16ca160750195aa137348cdb80ba9, NAME => 'TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.', STARTKEY => '', ENDKEY => ''} 2023-03-29 03:56:18,949 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 94b16ca160750195aa137348cdb80ba9 2023-03-29 03:56:18,949 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-29 03:56:18,949 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 94b16ca160750195aa137348cdb80ba9 2023-03-29 03:56:18,949 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 94b16ca160750195aa137348cdb80ba9 2023-03-29 03:56:18,950 INFO [StoreOpener-94b16ca160750195aa137348cdb80ba9-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 94b16ca160750195aa137348cdb80ba9 2023-03-29 03:56:18,951 DEBUG [StoreOpener-94b16ca160750195aa137348cdb80ba9-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A 2023-03-29 03:56:18,951 DEBUG [StoreOpener-94b16ca160750195aa137348cdb80ba9-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A 2023-03-29 03:56:18,951 INFO [StoreOpener-94b16ca160750195aa137348cdb80ba9-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-29 03:56:18,951 INFO [StoreOpener-94b16ca160750195aa137348cdb80ba9-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 94b16ca160750195aa137348cdb80ba9 columnFamilyName A 2023-03-29 03:56:18,952 INFO [StoreOpener-94b16ca160750195aa137348cdb80ba9-1] regionserver.HStore(310): Store=94b16ca160750195aa137348cdb80ba9/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-29 03:56:18,952 INFO [StoreOpener-94b16ca160750195aa137348cdb80ba9-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 94b16ca160750195aa137348cdb80ba9 2023-03-29 03:56:18,953 DEBUG [StoreOpener-94b16ca160750195aa137348cdb80ba9-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B 2023-03-29 03:56:18,953 DEBUG [StoreOpener-94b16ca160750195aa137348cdb80ba9-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B 2023-03-29 03:56:18,953 INFO [StoreOpener-94b16ca160750195aa137348cdb80ba9-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-29 03:56:18,953 INFO [StoreOpener-94b16ca160750195aa137348cdb80ba9-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 94b16ca160750195aa137348cdb80ba9 columnFamilyName B 2023-03-29 03:56:18,954 INFO [StoreOpener-94b16ca160750195aa137348cdb80ba9-1] regionserver.HStore(310): Store=94b16ca160750195aa137348cdb80ba9/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-29 03:56:18,954 INFO [StoreOpener-94b16ca160750195aa137348cdb80ba9-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 94b16ca160750195aa137348cdb80ba9 2023-03-29 03:56:18,954 DEBUG [StoreOpener-94b16ca160750195aa137348cdb80ba9-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C 2023-03-29 03:56:18,954 DEBUG [StoreOpener-94b16ca160750195aa137348cdb80ba9-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C 2023-03-29 03:56:18,954 INFO [StoreOpener-94b16ca160750195aa137348cdb80ba9-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-29 03:56:18,954 INFO [StoreOpener-94b16ca160750195aa137348cdb80ba9-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 94b16ca160750195aa137348cdb80ba9 columnFamilyName C 2023-03-29 03:56:18,955 INFO [StoreOpener-94b16ca160750195aa137348cdb80ba9-1] regionserver.HStore(310): Store=94b16ca160750195aa137348cdb80ba9/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-29 03:56:18,955 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:18,955 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9 2023-03-29 03:56:18,956 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9 2023-03-29 03:56:18,958 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-29 03:56:18,959 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 94b16ca160750195aa137348cdb80ba9 2023-03-29 03:56:18,959 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 94b16ca160750195aa137348cdb80ba9; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74080682, jitterRate=0.10388818383216858}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-29 03:56:18,959 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 94b16ca160750195aa137348cdb80ba9: 2023-03-29 03:56:18,960 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9., pid=61, masterSystemTime=1680062178946 2023-03-29 03:56:18,961 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:18,961 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:18,961 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=94b16ca160750195aa137348cdb80ba9, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:18,961 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680062178961"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1680062178961"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680062178961"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680062178961"}]},"ts":"1680062178961"} 2023-03-29 03:56:18,963 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=61, resume processing ppid=59 2023-03-29 03:56:18,963 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=61, ppid=59, state=SUCCESS; OpenRegionProcedure 94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 in 168 msec 2023-03-29 03:56:18,964 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=59, resume processing ppid=58 2023-03-29 03:56:18,964 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=59, ppid=58, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=94b16ca160750195aa137348cdb80ba9, REOPEN/MOVE in 483 msec 2023-03-29 03:56:18,966 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=58, resume processing ppid=57 2023-03-29 03:56:18,966 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=58, ppid=57, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 487 msec 2023-03-29 03:56:18,967 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=57, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 905 msec 2023-03-29 03:56:18,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=57 2023-03-29 03:56:18,968 DEBUG [Listener at localhost/35027] zookeeper.ReadOnlyZKClient(139): Connect 0x062b3caa to 127.0.0.1:52103 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-29 03:56:18,973 DEBUG [Listener at localhost/35027] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1d759aab, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-29 03:56:18,974 DEBUG [Listener at localhost/35027] zookeeper.ReadOnlyZKClient(139): Connect 0x66b5b1f3 to 127.0.0.1:52103 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-29 03:56:18,977 DEBUG [Listener at localhost/35027] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@39241fe3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-29 03:56:18,978 DEBUG [Listener at localhost/35027] zookeeper.ReadOnlyZKClient(139): Connect 0x544a4215 to 127.0.0.1:52103 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-29 03:56:18,982 DEBUG [Listener at localhost/35027] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1fbb5b46, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-29 03:56:18,982 DEBUG [Listener at localhost/35027] zookeeper.ReadOnlyZKClient(139): Connect 0x25baee3e to 127.0.0.1:52103 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-29 03:56:18,986 DEBUG [Listener at localhost/35027] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@23e65a76, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-29 03:56:18,986 DEBUG [Listener at localhost/35027] zookeeper.ReadOnlyZKClient(139): Connect 0x2c53bf96 to 127.0.0.1:52103 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-29 03:56:18,990 DEBUG [Listener at localhost/35027] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5e6a6f3c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-29 03:56:18,990 DEBUG [Listener at localhost/35027] zookeeper.ReadOnlyZKClient(139): Connect 0x0a470a3d to 127.0.0.1:52103 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-29 03:56:18,994 DEBUG [Listener at localhost/35027] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7d997c6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-29 03:56:18,995 DEBUG [Listener at localhost/35027] zookeeper.ReadOnlyZKClient(139): Connect 0x4bb72437 to 127.0.0.1:52103 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-29 03:56:19,003 DEBUG [Listener at localhost/35027] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7993d99e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-29 03:56:19,003 DEBUG [Listener at localhost/35027] zookeeper.ReadOnlyZKClient(139): Connect 0x3dca23c0 to 127.0.0.1:52103 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-29 03:56:19,010 DEBUG [Listener at localhost/35027] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4fae3ed2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-29 03:56:19,010 DEBUG [Listener at localhost/35027] zookeeper.ReadOnlyZKClient(139): Connect 0x62d52321 to 127.0.0.1:52103 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-29 03:56:19,016 DEBUG [Listener at localhost/35027] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@eedd898, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-29 03:56:19,016 DEBUG [Listener at localhost/35027] zookeeper.ReadOnlyZKClient(139): Connect 0x2a8bf279 to 127.0.0.1:52103 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-29 03:56:19,020 DEBUG [Listener at localhost/35027] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@618c0022, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-29 03:56:19,028 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:56:19,028 DEBUG [hconnection-0x479c9d7a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-29 03:56:19,033 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:37652, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-29 03:56:19,034 DEBUG [hconnection-0x376e389a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-29 03:56:19,036 DEBUG [hconnection-0xe4d4baa-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-29 03:56:19,036 DEBUG [hconnection-0x34950951-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-29 03:56:19,036 DEBUG [hconnection-0x5b3437cb-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-29 03:56:19,036 DEBUG [hconnection-0x2e7e35dd-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-29 03:56:19,037 DEBUG [hconnection-0x67e3cc2a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-29 03:56:19,037 DEBUG [hconnection-0x13ba9604-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-29 03:56:19,037 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:37670, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-29 03:56:19,037 DEBUG [hconnection-0x10eba27a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-29 03:56:19,038 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:37690, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-29 03:56:19,038 DEBUG [hconnection-0x2dcdeab0-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-29 03:56:19,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:56:19,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:56:19,040 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:56:19,040 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:19,040 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:56:19,040 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:56:19,042 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:37664, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-29 03:56:19,043 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:37666, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-29 03:56:19,043 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:37674, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-29 03:56:19,043 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:19,043 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:19,043 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:37724, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-29 03:56:19,043 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:37700, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-29 03:56:19,046 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:37712, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-29 03:56:19,048 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:19,048 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:19,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:19,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:19,048 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:19,048 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:56:19,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:19,049 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:19,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:56:19,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:19,049 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:56:19,049 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:56:19,049 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:56:19,049 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:19,049 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:56:19,050 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:56:19,050 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:56:19,050 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:19,050 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. started... 2023-03-29 03:56:19,053 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 94b16ca160750195aa137348cdb80ba9 3/3 column families, dataSize=6.71 KB heapSize=20.67 KB 2023-03-29 03:56:19,053 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94b16ca160750195aa137348cdb80ba9, store=A 2023-03-29 03:56:19,055 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:56:19,055 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94b16ca160750195aa137348cdb80ba9, store=B 2023-03-29 03:56:19,055 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:37734, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-29 03:56:19,055 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:56:19,055 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94b16ca160750195aa137348cdb80ba9, store=C 2023-03-29 03:56:19,055 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:56:19,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on 94b16ca160750195aa137348cdb80ba9 2023-03-29 03:56:19,060 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. as already flushing 2023-03-29 03:56:19,073 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680062179066, max timestamp=1680062179071Num uniques 859; 2023-03-29 03:56:19,073 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-29 03:56:19,075 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680062179066, max timestamp=1680062179071Num uniques 859; 2023-03-29 03:56:19,075 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-03-29 03:56:19,076 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680062179066, max timestamp=1680062179071Num uniques 859; 2023-03-29 03:56:19,076 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-03-29 03:56:19,084 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032990a2574918bf476da1fdb97ecfe74634 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032990a2574918bf476da1fdb97ecfe74634 2023-03-29 03:56:19,089 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680062179080, max timestamp=1680062179087Num uniques 1009; 2023-03-29 03:56:19,089 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46213, new segment data size=7688, suffix heap size=132904,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=168 2023-03-29 03:56:19,090 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1680062179080, max timestamp=1680062179087Num uniques 1018; 2023-03-29 03:56:19,090 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-03-29 03:56:19,091 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680062179080, max timestamp=1680062179088Num uniques 1009; 2023-03-29 03:56:19,091 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46213, new segment data size=7688, suffix heap size=132904,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=168 2023-03-29 03:56:19,091 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=8, memsize=2.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/A/f9f67b3b0d4b4ba3a97ede9e12e36d77 2023-03-29 03:56:19,108 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1680062179102, max timestamp=1680062179105Num uniques 1027; 2023-03-29 03:56:19,108 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-03-29 03:56:19,108 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1680062179102, max timestamp=1680062179105Num uniques 1077; 2023-03-29 03:56:19,108 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=8102, suffix heap size=139976,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=177 2023-03-29 03:56:19,108 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1680062179102, max timestamp=1680062179105Num uniques 1027; 2023-03-29 03:56:19,108 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-03-29 03:56:19,111 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.24 KB at sequenceid=8 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/B/7e4484834edd46fe92767797e5512b77 2023-03-29 03:56:19,124 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1680062179115, max timestamp=1680062179122Num uniques 1036; 2023-03-29 03:56:19,124 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=10806, suffix heap size=135712,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=236 2023-03-29 03:56:19,125 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1680062179115, max timestamp=1680062179122Num uniques 1036; 2023-03-29 03:56:19,125 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1680062179115, max timestamp=1680062179122Num uniques 1036; 2023-03-29 03:56:19,125 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=10806, suffix heap size=135712,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=236 2023-03-29 03:56:19,125 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=10806, suffix heap size=135712,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=236 2023-03-29 03:56:19,140 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1680062179133, max timestamp=1680062179138Num uniques 1095; 2023-03-29 03:56:19,140 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=11220, suffix heap size=141848,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=245 2023-03-29 03:56:19,140 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1680062179133, max timestamp=1680062179138Num uniques 1095; 2023-03-29 03:56:19,140 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=11220, suffix heap size=141848,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=245 2023-03-29 03:56:19,141 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1680062179133, max timestamp=1680062179138Num uniques 1095; 2023-03-29 03:56:19,141 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=11220, suffix heap size=141848,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=245 2023-03-29 03:56:19,147 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.24 KB at sequenceid=8 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/C/98d37c2abeea4853bad36ac1088dc8d6 2023-03-29 03:56:19,152 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/A/f9f67b3b0d4b4ba3a97ede9e12e36d77 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/f9f67b3b0d4b4ba3a97ede9e12e36d77 2023-03-29 03:56:19,155 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9334, totalHeapSize=21456, min timestamp=1680062179146, max timestamp=1680062179153Num uniques 1104; 2023-03-29 03:56:19,155 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=50554, new segment data size=9334, suffix heap size=142784,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=204 2023-03-29 03:56:19,155 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9334, totalHeapSize=21456, min timestamp=1680062179146, max timestamp=1680062179153Num uniques 1104; 2023-03-29 03:56:19,155 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/f9f67b3b0d4b4ba3a97ede9e12e36d77, entries=50, sequenceid=8, filesize=11.6 K 2023-03-29 03:56:19,155 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=50554, new segment data size=9334, suffix heap size=142784,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=204 2023-03-29 03:56:19,156 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9334, totalHeapSize=21456, min timestamp=1680062179146, max timestamp=1680062179153Num uniques 1104; 2023-03-29 03:56:19,156 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=50554, new segment data size=9334, suffix heap size=142784,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=204 2023-03-29 03:56:19,156 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/B/7e4484834edd46fe92767797e5512b77 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/7e4484834edd46fe92767797e5512b77 2023-03-29 03:56:19,159 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/7e4484834edd46fe92767797e5512b77, entries=50, sequenceid=8, filesize=7.0 K 2023-03-29 03:56:19,160 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/C/98d37c2abeea4853bad36ac1088dc8d6 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/98d37c2abeea4853bad36ac1088dc8d6 2023-03-29 03:56:19,164 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/98d37c2abeea4853bad36ac1088dc8d6, entries=50, sequenceid=8, filesize=7.0 K 2023-03-29 03:56:19,164 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~6.71 KB/6870, heapSize ~20.63 KB/21120, currentSize=100.63 KB/103050 for 94b16ca160750195aa137348cdb80ba9 in 111ms, sequenceid=8, compaction requested=false 2023-03-29 03:56:19,164 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-03-29 03:56:19,165 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 94b16ca160750195aa137348cdb80ba9: 2023-03-29 03:56:19,165 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:19,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on 94b16ca160750195aa137348cdb80ba9 2023-03-29 03:56:19,165 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 94b16ca160750195aa137348cdb80ba9 3/3 column families, dataSize=107.34 KB heapSize=301.08 KB 2023-03-29 03:56:19,165 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:56:19,165 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:56:19,165 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:19,165 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:56:19,165 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:56:19,166 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94b16ca160750195aa137348cdb80ba9, store=A 2023-03-29 03:56:19,167 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-29 03:56:19,167 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94b16ca160750195aa137348cdb80ba9, store=B 2023-03-29 03:56:19,167 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-29 03:56:19,167 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94b16ca160750195aa137348cdb80ba9, store=C 2023-03-29 03:56:19,167 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-29 03:56:19,167 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:19,167 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:19,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:19,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:19,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:19,167 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:19,167 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:56:19,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:19,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:19,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:19,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:19,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:19,169 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:56:19,169 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@763ea94f[Count = 0] remaining members to acquire global barrier 2023-03-29 03:56:19,169 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:56:19,169 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:19,171 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:19,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:19,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:19,171 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:56:19,171 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:56:19,171 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:56:19,172 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:56:19,172 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:19,172 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:56:19,172 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:19,173 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:56:19,173 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:56:19,173 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:56:19,173 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:56:19,173 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:56:19,173 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:56:19,173 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:56:19,176 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:19,176 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:19,176 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:19,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:19,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:19,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:19,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:19,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:19,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:19,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:19,176 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:19,176 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:19,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:19,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37674 deadline: 1680062239174, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:19,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:19,177 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:19,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:19,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37652 deadline: 1680062239175, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:19,177 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:19,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37670 deadline: 1680062239175, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:19,177 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:19,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:19,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37712 deadline: 1680062239177, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:19,178 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:19,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:19,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:19,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:19,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:19,180 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:19,180 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:19,181 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:19,181 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:19,181 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:19,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37666 deadline: 1680062239179, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:19,181 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:19,181 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:19,181 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:19,181 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:56:19,181 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:56:19,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:19,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:19,181 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:19,182 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:19,182 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:19,182 DEBUG [Thread-4303] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:56:19,182 DEBUG [Thread-4303] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:56:19,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:19,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:19,181 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:56:19,184 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230329df6580e982b5479195677a2a28fc165c to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230329df6580e982b5479195677a2a28fc165c 2023-03-29 03:56:19,190 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=127, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/A/8725b6616cf24accad5f453fe46e4ed2 2023-03-29 03:56:19,278 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:19,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37674 deadline: 1680062239278, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:19,280 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:19,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37670 deadline: 1680062239278, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:19,280 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:19,280 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:19,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37712 deadline: 1680062239278, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:19,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37652 deadline: 1680062239279, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:19,282 DEBUG [Thread-4303] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:56:19,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:56:19,283 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:19,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37666 deadline: 1680062239282, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:19,284 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:56:19,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:56:19,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:56:19,286 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:56:19,286 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:19,286 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:56:19,286 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:56:19,286 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:19,286 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:19,288 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:19,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:19,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:19,288 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:19,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:19,288 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:19,289 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:56:19,289 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:19,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:56:19,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:19,289 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:56:19,290 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:56:19,290 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:56:19,290 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:56:19,290 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:19,290 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:56:19,290 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:56:19,290 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:56:19,290 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:19,290 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. started... 2023-03-29 03:56:19,290 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. as already flushing 2023-03-29 03:56:19,290 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:19,290 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:56:19,290 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:56:19,290 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:19,290 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:56:19,290 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:56:19,292 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:19,292 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:19,292 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:19,292 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:56:19,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:19,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:19,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:19,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:19,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:19,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:19,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:19,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:19,294 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:56:19,294 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ec70f2a[Count = 0] remaining members to acquire global barrier 2023-03-29 03:56:19,294 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:56:19,294 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:19,297 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:19,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:19,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:19,297 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:56:19,297 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:56:19,297 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:56:19,297 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:19,297 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:56:19,298 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:19,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:19,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:19,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:19,298 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:56:19,298 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:19,298 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:56:19,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:19,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:19,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:19,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:19,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:19,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:19,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:19,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:56:19,301 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:56:19,301 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:56:19,301 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:56:19,301 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:56:19,301 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:56:19,301 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:56:19,317 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:19,317 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:19,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:19,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:19,317 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:19,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:19,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:19,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:19,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:19,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:19,318 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:19,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:19,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:19,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:19,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:19,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:19,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:19,319 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:19,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:19,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:19,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:19,322 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:19,322 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:19,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:19,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:19,322 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:56:19,322 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:56:19,322 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:19,322 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:19,322 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:19,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:19,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:19,322 DEBUG [Thread-4303] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:56:19,322 DEBUG [Thread-4303] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:56:19,322 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:19,322 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:19,322 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:19,322 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:19,322 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:19,422 DEBUG [Thread-4303] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:56:19,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:56:19,425 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:56:19,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:56:19,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:56:19,426 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:56:19,426 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:19,426 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:56:19,426 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:56:19,426 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:19,426 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:19,428 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:19,429 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:19,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:19,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:19,429 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:19,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:19,429 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:56:19,429 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:19,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:56:19,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:19,430 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:56:19,430 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:56:19,430 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:56:19,430 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:56:19,430 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:19,430 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:56:19,431 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:56:19,431 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:56:19,431 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:19,431 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. started... 2023-03-29 03:56:19,431 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. as already flushing 2023-03-29 03:56:19,431 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:19,431 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:56:19,432 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:56:19,432 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:19,432 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:56:19,432 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:56:19,433 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:19,433 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:19,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:19,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:19,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:19,434 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:19,434 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:56:19,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:19,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:19,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:19,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:19,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:19,435 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:56:19,435 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@55ac81ad[Count = 0] remaining members to acquire global barrier 2023-03-29 03:56:19,435 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:56:19,436 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:19,437 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:19,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:19,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:19,437 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:19,437 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:56:19,437 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:56:19,437 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:56:19,437 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:56:19,438 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:19,438 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:56:19,439 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:19,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:19,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:19,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:19,439 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:56:19,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:19,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:19,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:19,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:19,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:19,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:19,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:19,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:56:19,441 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:56:19,441 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:56:19,441 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:56:19,441 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:56:19,441 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:56:19,441 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:56:19,447 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:19,448 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:19,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:19,448 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:19,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:19,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:19,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:19,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:19,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:19,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:19,448 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:19,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:19,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:19,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:19,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:19,449 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:19,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:19,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:19,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:19,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:19,452 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:19,452 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:19,452 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:19,452 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:19,452 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:19,452 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:19,452 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:19,452 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:19,452 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:19,452 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:19,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:19,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:19,452 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:56:19,453 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:56:19,452 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:56:19,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:19,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:19,453 DEBUG [Thread-4303] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:56:19,453 DEBUG [Thread-4303] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:56:19,480 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:19,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37674 deadline: 1680062239480, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:19,482 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:19,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37670 deadline: 1680062239481, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:19,482 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:19,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37712 deadline: 1680062239481, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:19,482 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:19,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37652 deadline: 1680062239482, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:19,486 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:19,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37666 deadline: 1680062239486, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:19,553 DEBUG [Thread-4303] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:56:19,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:56:19,555 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:56:19,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:56:19,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:56:19,575 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:56:19,575 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:19,575 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:56:19,575 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:56:19,575 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:19,575 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:19,577 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:19,577 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:19,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:19,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:19,577 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:19,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:19,577 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:56:19,578 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:19,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:56:19,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:19,578 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:56:19,578 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:56:19,578 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:56:19,578 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:56:19,578 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:19,578 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:56:19,579 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:56:19,579 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:56:19,579 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:19,579 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. started... 2023-03-29 03:56:19,579 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. as already flushing 2023-03-29 03:56:19,579 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:19,579 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:56:19,579 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:56:19,579 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:19,579 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:56:19,579 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:56:19,580 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:19,581 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:19,581 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:19,581 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:56:19,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:19,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:19,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:19,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:19,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:19,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:19,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:19,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:19,583 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:56:19,583 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@52b31abb[Count = 0] remaining members to acquire global barrier 2023-03-29 03:56:19,583 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:56:19,583 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:19,585 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:19,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:19,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:19,585 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:56:19,585 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:56:19,585 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:56:19,585 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:19,585 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:56:19,587 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:56:19,587 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:19,587 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:19,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:19,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:19,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:19,587 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:56:19,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:19,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:19,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:19,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:19,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:19,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:19,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:19,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:56:19,589 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:56:19,589 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:56:19,589 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:56:19,590 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:56:19,590 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:56:19,590 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:56:19,591 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:19,591 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:19,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:19,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:19,591 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:19,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:19,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:19,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:19,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:19,592 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:19,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:19,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:19,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:19,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:19,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:19,593 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:19,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:19,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:19,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:19,595 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:19,595 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:19,595 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:19,595 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:19,595 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:19,595 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:19,595 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:19,595 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:56:19,595 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:56:19,595 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:19,595 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:19,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:19,596 DEBUG [Thread-4303] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:56:19,595 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:19,596 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:56:19,596 DEBUG [Thread-4303] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:56:19,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:19,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:19,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:19,600 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=127 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/B/284d397ea2b44c92ab4085b6b0e584e3 2023-03-29 03:56:19,610 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=127 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/C/a0bd07e926d5475c8fbef239de91c279 2023-03-29 03:56:19,613 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/A/8725b6616cf24accad5f453fe46e4ed2 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/8725b6616cf24accad5f453fe46e4ed2 2023-03-29 03:56:19,617 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/8725b6616cf24accad5f453fe46e4ed2, entries=150, sequenceid=127, filesize=25.1 K 2023-03-29 03:56:19,617 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/B/284d397ea2b44c92ab4085b6b0e584e3 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/284d397ea2b44c92ab4085b6b0e584e3 2023-03-29 03:56:19,621 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/284d397ea2b44c92ab4085b6b0e584e3, entries=150, sequenceid=127, filesize=11.6 K 2023-03-29 03:56:19,622 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/C/a0bd07e926d5475c8fbef239de91c279 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/a0bd07e926d5475c8fbef239de91c279 2023-03-29 03:56:19,630 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/a0bd07e926d5475c8fbef239de91c279, entries=150, sequenceid=127, filesize=11.6 K 2023-03-29 03:56:19,631 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~320.95 KB/328656, currentSize=67.09 KB/68700 for 94b16ca160750195aa137348cdb80ba9 in 466ms, sequenceid=127, compaction requested=false 2023-03-29 03:56:19,632 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 94b16ca160750195aa137348cdb80ba9: 2023-03-29 03:56:19,696 DEBUG [Thread-4303] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:56:19,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:56:19,704 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:56:19,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:56:19,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:56:19,706 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:56:19,706 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:19,706 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:56:19,706 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:56:19,706 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:19,706 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:19,709 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:19,709 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:19,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:19,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:19,710 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:19,710 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:56:19,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:19,710 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:19,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:56:19,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:19,710 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:56:19,711 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:56:19,711 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:56:19,711 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:56:19,711 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:19,711 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:56:19,711 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:56:19,711 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:56:19,711 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:19,711 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. started... 2023-03-29 03:56:19,712 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 94b16ca160750195aa137348cdb80ba9 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-29 03:56:19,712 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94b16ca160750195aa137348cdb80ba9, store=A 2023-03-29 03:56:19,712 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:56:19,712 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94b16ca160750195aa137348cdb80ba9, store=B 2023-03-29 03:56:19,712 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:56:19,712 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94b16ca160750195aa137348cdb80ba9, store=C 2023-03-29 03:56:19,712 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:56:19,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on 94b16ca160750195aa137348cdb80ba9 2023-03-29 03:56:19,785 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. as already flushing 2023-03-29 03:56:19,802 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:19,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37652 deadline: 1680062239799, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:19,802 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:19,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37670 deadline: 1680062239799, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:19,802 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:19,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37674 deadline: 1680062239800, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:19,803 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:19,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37666 deadline: 1680062239800, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:19,803 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:19,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37712 deadline: 1680062239801, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:19,904 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:19,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37652 deadline: 1680062239903, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:19,905 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:19,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37670 deadline: 1680062239903, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:19,905 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:19,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37674 deadline: 1680062239903, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:19,905 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:19,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37666 deadline: 1680062239904, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:19,905 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:19,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37712 deadline: 1680062239904, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:20,107 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:20,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37670 deadline: 1680062240106, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:20,108 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:20,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37666 deadline: 1680062240106, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:20,108 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:20,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37712 deadline: 1680062240106, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:20,108 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:20,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37652 deadline: 1680062240106, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:20,108 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:20,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37674 deadline: 1680062240107, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:20,134 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230329957670930d8843c5a8b52553f22d629e to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230329957670930d8843c5a8b52553f22d629e 2023-03-29 03:56:20,140 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=140, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/A/183379056af8462fbc92a6b88d79b344 2023-03-29 03:56:20,151 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=140 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/B/344bc1207b904bf6a57af2e3a1f2c3cd 2023-03-29 03:56:20,159 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=140 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/C/bc6336b3978840f4ab53bb5d285b6b7d 2023-03-29 03:56:20,162 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/A/183379056af8462fbc92a6b88d79b344 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/183379056af8462fbc92a6b88d79b344 2023-03-29 03:56:20,166 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/183379056af8462fbc92a6b88d79b344, entries=100, sequenceid=140, filesize=18.5 K 2023-03-29 03:56:20,167 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/B/344bc1207b904bf6a57af2e3a1f2c3cd as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/344bc1207b904bf6a57af2e3a1f2c3cd 2023-03-29 03:56:20,170 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/344bc1207b904bf6a57af2e3a1f2c3cd, entries=100, sequenceid=140, filesize=9.4 K 2023-03-29 03:56:20,171 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/C/bc6336b3978840f4ab53bb5d285b6b7d as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/bc6336b3978840f4ab53bb5d285b6b7d 2023-03-29 03:56:20,173 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/bc6336b3978840f4ab53bb5d285b6b7d, entries=100, sequenceid=140, filesize=9.4 K 2023-03-29 03:56:20,174 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=114.05 KB/116790 for 94b16ca160750195aa137348cdb80ba9 in 462ms, sequenceid=140, compaction requested=true 2023-03-29 03:56:20,174 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 94b16ca160750195aa137348cdb80ba9: 2023-03-29 03:56:20,174 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:20,174 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:56:20,174 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:56:20,175 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:20,175 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:56:20,175 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:56:20,176 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:20,176 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:20,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:20,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:20,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:20,177 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:20,177 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:56:20,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:20,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:20,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:20,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:20,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:20,178 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:56:20,179 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5144a573[Count = 0] remaining members to acquire global barrier 2023-03-29 03:56:20,179 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:56:20,179 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:20,180 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:20,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:20,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:20,180 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:56:20,180 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:56:20,180 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:20,180 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:56:20,180 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:56:20,181 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:20,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:20,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:20,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:20,182 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:56:20,182 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:20,182 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:56:20,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:20,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:20,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:20,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:20,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:20,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:20,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:20,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:56:20,184 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:56:20,184 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:56:20,184 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:56:20,184 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:56:20,184 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:56:20,184 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:56:20,189 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:20,189 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:20,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:20,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:20,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:20,189 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:20,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:20,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:20,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:20,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:20,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:20,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:20,190 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:20,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:20,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:20,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:20,191 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:20,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:20,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:20,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:20,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:20,194 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:20,194 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:20,194 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:20,194 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:20,194 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:20,194 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:20,194 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:20,194 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:20,194 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:20,194 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:20,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:20,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:20,194 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:56:20,195 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:56:20,194 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:56:20,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:20,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:20,195 DEBUG [Thread-4303] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:56:20,195 DEBUG [Thread-4303] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:56:20,295 DEBUG [Thread-4303] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:56:20,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:56:20,297 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:56:20,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:56:20,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:56:20,299 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:56:20,299 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:20,299 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:56:20,299 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:56:20,299 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:20,299 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:20,301 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:20,301 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:20,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:20,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:20,301 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:20,301 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:56:20,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:20,302 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:20,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:56:20,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:20,302 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:56:20,302 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:56:20,302 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:56:20,302 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:56:20,303 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:20,303 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:56:20,303 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:56:20,303 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:56:20,303 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:20,303 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. started... 2023-03-29 03:56:20,303 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 94b16ca160750195aa137348cdb80ba9 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-29 03:56:20,303 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94b16ca160750195aa137348cdb80ba9, store=A 2023-03-29 03:56:20,303 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:56:20,303 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94b16ca160750195aa137348cdb80ba9, store=B 2023-03-29 03:56:20,303 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:56:20,303 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94b16ca160750195aa137348cdb80ba9, store=C 2023-03-29 03:56:20,303 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:56:20,325 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303293b5d44b2889e49f6b0d4985b4cb11c29 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303293b5d44b2889e49f6b0d4985b4cb11c29 2023-03-29 03:56:20,333 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=160, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/A/d414b1190d624823baac2744acbe5edf 2023-03-29 03:56:20,350 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=160 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/B/cd4aef3dc6414c10bf2b93115d929229 2023-03-29 03:56:20,365 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=160 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/C/f2ca8378cb984ac18483d4740747566b 2023-03-29 03:56:20,370 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/A/d414b1190d624823baac2744acbe5edf as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/d414b1190d624823baac2744acbe5edf 2023-03-29 03:56:20,373 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/d414b1190d624823baac2744acbe5edf, entries=150, sequenceid=160, filesize=25.3 K 2023-03-29 03:56:20,374 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/B/cd4aef3dc6414c10bf2b93115d929229 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/cd4aef3dc6414c10bf2b93115d929229 2023-03-29 03:56:20,377 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/cd4aef3dc6414c10bf2b93115d929229, entries=150, sequenceid=160, filesize=11.8 K 2023-03-29 03:56:20,378 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/C/f2ca8378cb984ac18483d4740747566b as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/f2ca8378cb984ac18483d4740747566b 2023-03-29 03:56:20,381 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/f2ca8378cb984ac18483d4740747566b, entries=150, sequenceid=160, filesize=11.8 K 2023-03-29 03:56:20,382 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=0 B/0 for 94b16ca160750195aa137348cdb80ba9 in 79ms, sequenceid=160, compaction requested=true 2023-03-29 03:56:20,382 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 94b16ca160750195aa137348cdb80ba9: 2023-03-29 03:56:20,382 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:20,382 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:56:20,382 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:56:20,382 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:20,382 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:56:20,382 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:56:20,385 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:20,385 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:20,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:20,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:20,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:20,386 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:20,386 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:56:20,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:20,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:20,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:20,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:20,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:20,387 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:56:20,387 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@371c4e2e[Count = 0] remaining members to acquire global barrier 2023-03-29 03:56:20,387 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:56:20,387 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:20,389 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:20,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:20,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:20,390 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:56:20,390 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:56:20,390 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:20,390 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:56:20,390 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:56:20,391 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:20,391 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:56:20,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:20,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:20,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:20,391 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:20,391 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:56:20,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:20,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:20,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:20,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:20,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:20,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:20,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:20,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:56:20,394 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:56:20,394 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:56:20,394 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:56:20,394 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:56:20,394 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:56:20,394 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:56:20,395 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:20,396 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:20,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:20,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:20,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:20,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:20,396 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:20,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:20,396 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:20,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:20,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:20,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:20,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:20,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:20,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:20,397 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:20,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:20,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:20,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:20,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:20,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:20,403 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:20,403 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:20,403 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:20,403 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:20,403 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:56:20,403 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:56:20,403 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:20,403 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:20,403 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:20,403 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:20,403 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:20,403 DEBUG [Thread-4303] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:56:20,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:20,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:20,403 DEBUG [Thread-4303] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:56:20,403 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:56:20,403 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:20,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:20,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:20,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on 94b16ca160750195aa137348cdb80ba9 2023-03-29 03:56:20,431 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 94b16ca160750195aa137348cdb80ba9 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-29 03:56:20,432 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94b16ca160750195aa137348cdb80ba9, store=A 2023-03-29 03:56:20,432 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:56:20,432 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94b16ca160750195aa137348cdb80ba9, store=B 2023-03-29 03:56:20,432 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:56:20,432 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94b16ca160750195aa137348cdb80ba9, store=C 2023-03-29 03:56:20,432 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:56:20,450 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032960b691a4c8404fed80aa2b6db0e3fcc2 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032960b691a4c8404fed80aa2b6db0e3fcc2 2023-03-29 03:56:20,455 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:20,456 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:20,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37712 deadline: 1680062240450, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:20,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37670 deadline: 1680062240449, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:20,456 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:20,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37652 deadline: 1680062240450, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:20,456 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:20,457 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:20,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37666 deadline: 1680062240451, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:20,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37674 deadline: 1680062240451, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:20,504 DEBUG [Thread-4303] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:56:20,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:56:20,505 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:56:20,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:56:20,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:56:20,506 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:56:20,506 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:20,506 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:56:20,506 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:56:20,507 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:20,507 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:20,508 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:20,508 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:20,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:20,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:20,508 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:20,509 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:56:20,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:20,509 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:20,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:56:20,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:20,509 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:56:20,509 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:56:20,510 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:56:20,510 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:56:20,510 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:20,510 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:56:20,510 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:56:20,510 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:56:20,510 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:20,510 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. started... 2023-03-29 03:56:20,510 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. as already flushing 2023-03-29 03:56:20,510 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:20,510 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:56:20,510 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:56:20,510 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:20,510 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:56:20,510 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:56:20,511 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:20,511 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:20,512 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:20,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:20,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:20,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:20,512 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:56:20,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:20,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:20,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:20,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:20,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:20,513 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:56:20,513 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@60c1089[Count = 0] remaining members to acquire global barrier 2023-03-29 03:56:20,513 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:56:20,513 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:20,514 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:20,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:20,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:20,515 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:20,515 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:56:20,515 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:56:20,515 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:56:20,515 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:56:20,517 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:56:20,517 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:20,517 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:20,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:20,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:20,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:20,518 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:56:20,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:20,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:20,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:20,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:20,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:20,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:20,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:20,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:56:20,521 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:56:20,521 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:56:20,521 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:56:20,521 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:56:20,521 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:56:20,521 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:56:20,523 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:20,523 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:20,523 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:20,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:20,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:20,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:20,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:20,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:20,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:20,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:20,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:20,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:20,523 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:20,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:20,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:20,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:20,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:20,525 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:20,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:20,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:20,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:20,528 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:20,528 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:20,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:20,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:20,528 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:20,528 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:20,528 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:20,528 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:20,528 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:20,528 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:20,528 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:20,528 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:56:20,528 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:56:20,529 DEBUG [Thread-4303] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:56:20,529 DEBUG [Thread-4303] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:56:20,528 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:20,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:20,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:20,559 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:20,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37712 deadline: 1680062240557, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:20,561 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:20,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37674 deadline: 1680062240558, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:20,561 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:20,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37670 deadline: 1680062240558, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:20,561 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:20,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37652 deadline: 1680062240558, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:20,562 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:20,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37666 deadline: 1680062240559, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:20,629 DEBUG [Thread-4303] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:56:20,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:56:20,632 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:56:20,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:56:20,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:56:20,634 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:56:20,634 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:20,634 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:56:20,634 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:56:20,634 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:20,634 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:20,635 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:20,636 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:20,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:20,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:20,636 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:20,636 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:56:20,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:20,636 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:20,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:56:20,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:20,637 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:56:20,637 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:56:20,637 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:56:20,637 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:56:20,637 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:20,637 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:56:20,638 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:56:20,638 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:56:20,638 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:20,638 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. started... 2023-03-29 03:56:20,638 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. as already flushing 2023-03-29 03:56:20,638 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:20,638 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:56:20,638 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:56:20,638 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:20,638 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:56:20,638 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:56:20,644 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:20,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:20,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:20,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:20,644 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:20,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:20,644 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:20,644 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:56:20,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:20,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:20,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:20,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:20,645 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:56:20,645 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c20b431[Count = 0] remaining members to acquire global barrier 2023-03-29 03:56:20,645 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:56:20,646 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:20,647 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:20,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:20,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:20,647 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:56:20,647 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:20,647 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:56:20,647 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:56:20,647 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:56:20,649 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:20,649 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:56:20,649 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:20,649 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:56:20,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:20,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:20,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:20,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:20,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:20,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:20,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:20,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:20,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:20,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:20,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:56:20,651 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:56:20,651 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:56:20,652 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:56:20,652 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:56:20,652 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:56:20,652 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:56:20,653 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:20,653 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:20,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:20,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:20,653 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:20,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:20,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:20,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:20,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:20,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:20,654 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:20,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:20,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:20,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:20,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:20,655 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:20,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:20,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:20,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:20,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:20,658 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:20,658 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:20,658 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:20,658 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:20,658 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:20,658 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:20,658 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:20,659 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:20,659 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:20,659 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:20,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:20,659 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:56:20,659 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:56:20,659 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:56:20,659 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:20,659 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:20,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:20,659 DEBUG [Thread-4303] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:56:20,659 DEBUG [Thread-4303] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:56:20,759 DEBUG [Thread-4303] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:56:20,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:56:20,761 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:56:20,761 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:20,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37712 deadline: 1680062240761, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:20,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:56:20,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:56:20,763 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:56:20,763 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:20,763 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:56:20,763 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:56:20,763 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:20,763 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:20,764 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:20,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:20,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:20,765 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:20,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:20,765 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:20,765 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:56:20,765 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:20,766 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:20,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:56:20,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:20,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37674 deadline: 1680062240763, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:20,766 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:56:20,766 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:20,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37652 deadline: 1680062240763, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:20,766 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:20,766 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:56:20,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37666 deadline: 1680062240764, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:20,766 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:56:20,767 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:56:20,767 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:20,767 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:56:20,767 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:56:20,767 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:56:20,767 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:20,767 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. started... 2023-03-29 03:56:20,767 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. as already flushing 2023-03-29 03:56:20,767 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:20,767 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:56:20,767 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:56:20,767 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:20,767 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:56:20,767 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:56:20,769 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:20,769 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:20,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:20,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:20,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:20,769 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:20,770 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:56:20,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:20,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:20,770 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:20,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37670 deadline: 1680062240768, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:20,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:20,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:20,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:20,771 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:56:20,771 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@383beffb[Count = 0] remaining members to acquire global barrier 2023-03-29 03:56:20,771 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:56:20,771 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:20,772 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:20,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:20,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:20,772 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:56:20,772 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:56:20,773 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:56:20,773 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:20,773 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:56:20,776 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:20,776 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:56:20,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:20,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:20,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:20,776 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:20,776 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:56:20,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:20,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:20,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:20,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:20,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:20,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:20,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:20,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:56:20,778 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:56:20,778 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:56:20,778 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:56:20,778 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:56:20,778 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:56:20,778 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:56:20,780 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:20,780 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:20,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:20,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:20,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:20,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:20,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:20,780 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:20,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:20,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:20,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:20,780 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:20,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:20,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:20,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:20,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:20,782 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:20,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:20,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:20,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:20,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:20,786 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:20,786 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:20,786 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:20,786 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:56:20,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:20,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:20,786 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:56:20,786 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:56:20,786 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:20,787 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:20,787 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:20,787 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:20,787 DEBUG [Thread-4303] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:56:20,787 DEBUG [Thread-4303] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:56:20,787 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:20,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:20,787 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:20,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:20,787 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:20,860 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=172, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/A/88fdd8f90adb43a39a41aeb6f5d59e69 2023-03-29 03:56:20,869 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=172 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/B/9ee0db13a4d54259b15727e487810f0d 2023-03-29 03:56:20,887 DEBUG [Thread-4303] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:56:20,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:56:20,889 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:56:20,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:56:20,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:56:20,890 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:56:20,890 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:20,890 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:56:20,890 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:56:20,891 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:20,891 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:20,893 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:20,893 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:20,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:20,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:20,893 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:20,893 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:56:20,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:20,894 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:20,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:56:20,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:20,894 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:56:20,894 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:56:20,894 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:56:20,894 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:56:20,894 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:20,895 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:56:20,895 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:56:20,895 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:56:20,895 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:20,895 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. started... 2023-03-29 03:56:20,895 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. as already flushing 2023-03-29 03:56:20,895 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:20,895 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:56:20,895 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:56:20,895 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:20,895 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:56:20,895 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:56:20,897 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:20,897 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:20,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:20,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:20,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:20,897 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:20,897 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:56:20,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:20,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:20,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:20,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:20,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:20,899 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:56:20,899 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6d03bbec[Count = 0] remaining members to acquire global barrier 2023-03-29 03:56:20,899 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:56:20,899 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:20,900 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:20,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:20,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:20,900 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:56:20,900 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:56:20,900 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:56:20,900 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:20,901 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:56:20,902 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:20,902 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:56:20,902 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:20,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:20,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:20,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:20,902 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:56:20,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:20,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:20,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:20,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:20,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:20,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:20,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:20,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:56:20,904 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:56:20,904 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:56:20,904 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:56:20,904 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:56:20,904 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:56:20,904 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:56:20,907 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:20,907 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:20,907 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:20,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:20,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:20,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:20,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:20,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:20,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:20,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:20,908 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:20,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:20,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:20,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:20,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:20,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:20,909 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:20,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:20,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:20,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:20,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:20,912 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:20,912 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:20,912 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:20,912 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:20,912 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:20,912 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:20,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:20,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:20,913 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:20,913 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:20,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:20,913 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:20,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:20,913 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:20,913 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:56:20,913 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:56:20,913 DEBUG [Thread-4303] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:56:20,913 DEBUG [Thread-4303] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:56:20,913 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:56:21,013 DEBUG [Thread-4303] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:56:21,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:56:21,015 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:56:21,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:56:21,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:56:21,017 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:56:21,017 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:21,017 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:56:21,017 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:56:21,017 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:21,017 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:21,019 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:21,019 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:21,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:21,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:21,019 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:21,019 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:56:21,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:21,020 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:21,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:56:21,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:21,020 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:56:21,021 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:56:21,021 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:56:21,021 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:56:21,021 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:21,021 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:56:21,021 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:56:21,021 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:56:21,021 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:21,021 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. started... 2023-03-29 03:56:21,021 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. as already flushing 2023-03-29 03:56:21,021 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:21,021 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:56:21,021 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:56:21,021 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:21,022 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:56:21,022 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:56:21,023 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:21,023 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:21,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:21,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:21,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:21,024 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:21,024 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:56:21,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:21,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:21,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:21,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:21,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:21,026 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:56:21,026 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@bed669d[Count = 0] remaining members to acquire global barrier 2023-03-29 03:56:21,026 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:56:21,026 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:21,028 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:21,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:21,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:21,028 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:56:21,028 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:56:21,028 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:56:21,029 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:21,029 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:56:21,030 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:21,030 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:56:21,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:21,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:21,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:21,030 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:21,030 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:56:21,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:21,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:21,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:21,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:21,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:21,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:21,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:21,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:56:21,033 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:56:21,033 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:56:21,033 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:56:21,033 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:56:21,033 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:56:21,033 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:56:21,035 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:21,035 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:21,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:21,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:21,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:21,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:21,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:21,035 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:21,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:21,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:21,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:21,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:21,036 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:21,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:21,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:21,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:21,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:21,038 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:21,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:21,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:21,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:21,040 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:21,040 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:21,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:21,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:21,040 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:21,041 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:21,041 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:21,041 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:21,041 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:56:21,041 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:56:21,041 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:21,041 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:21,041 DEBUG [Thread-4303] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:56:21,041 DEBUG [Thread-4303] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:56:21,041 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:21,041 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:21,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:21,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:21,066 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:21,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37712 deadline: 1680062241065, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:21,070 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:21,070 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:21,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37674 deadline: 1680062241067, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:21,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37652 deadline: 1680062241067, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:21,070 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:21,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37666 deadline: 1680062241068, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:21,074 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:21,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37670 deadline: 1680062241073, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:21,141 DEBUG [Thread-4303] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:56:21,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:56:21,143 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:56:21,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:56:21,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:56:21,145 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:56:21,145 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:21,145 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:56:21,145 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:56:21,145 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:21,145 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:21,146 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:21,146 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:21,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:21,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:21,147 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:21,147 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:56:21,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:21,147 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:21,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:56:21,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:21,148 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:56:21,148 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:56:21,148 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:56:21,148 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:56:21,148 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:21,148 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:56:21,148 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:56:21,148 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:56:21,148 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:21,148 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. started... 2023-03-29 03:56:21,149 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. as already flushing 2023-03-29 03:56:21,149 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:21,149 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:56:21,149 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:56:21,149 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:21,149 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:56:21,149 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:56:21,150 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:21,150 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:21,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:21,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:21,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:21,151 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:21,151 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:56:21,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:21,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:21,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:21,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:21,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:21,152 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:56:21,152 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4225daed[Count = 0] remaining members to acquire global barrier 2023-03-29 03:56:21,152 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:56:21,152 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:21,153 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:21,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:21,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:21,153 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:56:21,153 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:56:21,153 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:56:21,153 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:21,153 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:56:21,156 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:21,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:21,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:21,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:21,156 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:56:21,156 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:21,157 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:56:21,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:21,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:21,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:21,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:21,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:21,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:21,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:21,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:56:21,159 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:56:21,159 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:56:21,159 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:56:21,159 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:56:21,159 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:56:21,159 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:56:21,160 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:21,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:21,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:21,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:21,160 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:21,160 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:21,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:21,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:21,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:21,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:21,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:21,161 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:21,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:21,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:21,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:21,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:21,162 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:21,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:21,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:21,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:21,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:21,164 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:21,165 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:21,165 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:21,165 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:21,165 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:21,165 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:21,165 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:56:21,165 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:56:21,165 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:21,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:21,165 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:21,165 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:21,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:21,165 DEBUG [Thread-4303] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:56:21,165 DEBUG [Thread-4303] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:56:21,165 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:21,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:21,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:21,265 DEBUG [Thread-4303] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:56:21,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:56:21,267 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:56:21,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:56:21,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:56:21,269 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:56:21,269 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:21,269 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:56:21,269 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:56:21,269 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:21,269 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:21,271 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:21,271 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:21,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:21,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:21,271 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:21,271 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:56:21,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:21,271 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:21,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:56:21,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:21,272 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:56:21,272 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:56:21,272 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:56:21,272 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:56:21,272 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:21,272 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:56:21,276 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:56:21,276 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:56:21,276 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:21,276 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. started... 2023-03-29 03:56:21,276 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. as already flushing 2023-03-29 03:56:21,276 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:21,276 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:56:21,276 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:56:21,276 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:21,276 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:56:21,276 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:56:21,281 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=172 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/C/67394449c9d94d699b3209ea6ff531ce 2023-03-29 03:56:21,282 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:21,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:21,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:21,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:21,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:21,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:21,282 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:21,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:21,284 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:21,284 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:56:21,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:21,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:21,285 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:56:21,285 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/A/88fdd8f90adb43a39a41aeb6f5d59e69 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/88fdd8f90adb43a39a41aeb6f5d59e69 2023-03-29 03:56:21,285 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:56:21,286 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:21,285 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2586a165[Count = 0] remaining members to acquire global barrier 2023-03-29 03:56:21,287 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:21,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:21,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:21,287 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:56:21,287 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:56:21,287 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:56:21,288 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:21,288 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:56:21,289 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:56:21,289 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:21,289 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:56:21,289 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:21,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:21,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:21,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:21,289 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/88fdd8f90adb43a39a41aeb6f5d59e69, entries=450, sequenceid=172, filesize=66.3 K 2023-03-29 03:56:21,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:21,290 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/B/9ee0db13a4d54259b15727e487810f0d as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/9ee0db13a4d54259b15727e487810f0d 2023-03-29 03:56:21,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:21,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:21,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:21,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:21,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:21,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:21,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:56:21,292 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:56:21,292 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:56:21,292 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:56:21,292 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:56:21,292 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:56:21,292 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:56:21,293 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/9ee0db13a4d54259b15727e487810f0d, entries=150, sequenceid=172, filesize=11.8 K 2023-03-29 03:56:21,294 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:21,294 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:21,294 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:21,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:21,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:21,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:21,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:21,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:21,295 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/C/67394449c9d94d699b3209ea6ff531ce as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/67394449c9d94d699b3209ea6ff531ce 2023-03-29 03:56:21,295 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:21,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:21,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:21,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:21,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:21,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:21,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:21,296 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:21,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:21,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:21,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:21,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:21,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:21,299 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/67394449c9d94d699b3209ea6ff531ce, entries=150, sequenceid=172, filesize=11.8 K 2023-03-29 03:56:21,300 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 94b16ca160750195aa137348cdb80ba9 in 869ms, sequenceid=172, compaction requested=true 2023-03-29 03:56:21,300 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 94b16ca160750195aa137348cdb80ba9: 2023-03-29 03:56:21,300 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:56:21,300 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:56:21,300 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-29 03:56:21,300 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-29 03:56:21,300 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:56:21,301 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:21,301 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:21,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:21,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:21,301 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:21,301 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:21,301 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:21,301 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:21,301 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:21,301 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:21,301 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:21,301 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:56:21,301 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:56:21,301 DEBUG [Thread-4303] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:56:21,302 DEBUG [Thread-4303] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:56:21,301 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:21,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:21,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:21,302 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 150240 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-29 03:56:21,302 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1912): 94b16ca160750195aa137348cdb80ba9/A is initiating minor compaction (all files) 2023-03-29 03:56:21,302 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 94b16ca160750195aa137348cdb80ba9/A in TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:21,302 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/f9f67b3b0d4b4ba3a97ede9e12e36d77, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/8725b6616cf24accad5f453fe46e4ed2, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/183379056af8462fbc92a6b88d79b344, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/d414b1190d624823baac2744acbe5edf, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/88fdd8f90adb43a39a41aeb6f5d59e69] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp, totalSize=146.7 K 2023-03-29 03:56:21,303 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:56:21,303 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting f9f67b3b0d4b4ba3a97ede9e12e36d77, keycount=50, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=8, earliestPutTs=1680062179036 2023-03-29 03:56:21,303 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 52800 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-29 03:56:21,303 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 8725b6616cf24accad5f453fe46e4ed2, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=127, earliestPutTs=1680062179153 2023-03-29 03:56:21,303 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1912): 94b16ca160750195aa137348cdb80ba9/B is initiating minor compaction (all files) 2023-03-29 03:56:21,303 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 94b16ca160750195aa137348cdb80ba9/B in TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:21,303 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/7e4484834edd46fe92767797e5512b77, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/284d397ea2b44c92ab4085b6b0e584e3, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/344bc1207b904bf6a57af2e3a1f2c3cd, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/cd4aef3dc6414c10bf2b93115d929229, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/9ee0db13a4d54259b15727e487810f0d] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp, totalSize=51.6 K 2023-03-29 03:56:21,303 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 183379056af8462fbc92a6b88d79b344, keycount=100, bloomtype=ROW, size=18.5 K, encoding=NONE, compression=NONE, seqNum=140, earliestPutTs=1680062179171 2023-03-29 03:56:21,304 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 7e4484834edd46fe92767797e5512b77, keycount=50, bloomtype=ROW, size=7.0 K, encoding=NONE, compression=NONE, seqNum=8, earliestPutTs=1680062179036 2023-03-29 03:56:21,304 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting d414b1190d624823baac2744acbe5edf, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=160, earliestPutTs=1680062179799 2023-03-29 03:56:21,304 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 284d397ea2b44c92ab4085b6b0e584e3, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=127, earliestPutTs=1680062179153 2023-03-29 03:56:21,304 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 88fdd8f90adb43a39a41aeb6f5d59e69, keycount=450, bloomtype=ROW, size=66.3 K, encoding=NONE, compression=NONE, seqNum=172, earliestPutTs=1680062180418 2023-03-29 03:56:21,305 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 344bc1207b904bf6a57af2e3a1f2c3cd, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=140, earliestPutTs=1680062179171 2023-03-29 03:56:21,305 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting cd4aef3dc6414c10bf2b93115d929229, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=160, earliestPutTs=1680062179799 2023-03-29 03:56:21,306 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 9ee0db13a4d54259b15727e487810f0d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=172, earliestPutTs=1680062180421 2023-03-29 03:56:21,314 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] throttle.PressureAwareThroughputController(145): 94b16ca160750195aa137348cdb80ba9#B#compaction#604 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:56:21,317 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] throttle.PressureAwareThroughputController(145): 94b16ca160750195aa137348cdb80ba9#A#compaction#603 average throughput is 3.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:56:21,354 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/A/2c7bb4c01e1346e7b210bbc1a227a40f as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/2c7bb4c01e1346e7b210bbc1a227a40f 2023-03-29 03:56:21,357 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 94b16ca160750195aa137348cdb80ba9/A of 94b16ca160750195aa137348cdb80ba9 into 2c7bb4c01e1346e7b210bbc1a227a40f(size=25.5 K), total size for store is 25.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:56:21,358 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 94b16ca160750195aa137348cdb80ba9: 2023-03-29 03:56:21,358 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9., storeName=94b16ca160750195aa137348cdb80ba9/A, priority=11, startTime=1680062181300; duration=0sec 2023-03-29 03:56:21,358 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:56:21,358 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-29 03:56:21,359 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 52800 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-29 03:56:21,359 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1912): 94b16ca160750195aa137348cdb80ba9/C is initiating minor compaction (all files) 2023-03-29 03:56:21,359 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 94b16ca160750195aa137348cdb80ba9/C in TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:21,360 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/98d37c2abeea4853bad36ac1088dc8d6, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/a0bd07e926d5475c8fbef239de91c279, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/bc6336b3978840f4ab53bb5d285b6b7d, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/f2ca8378cb984ac18483d4740747566b, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/67394449c9d94d699b3209ea6ff531ce] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp, totalSize=51.6 K 2023-03-29 03:56:21,362 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 98d37c2abeea4853bad36ac1088dc8d6, keycount=50, bloomtype=ROW, size=7.0 K, encoding=NONE, compression=NONE, seqNum=8, earliestPutTs=1680062179036 2023-03-29 03:56:21,362 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting a0bd07e926d5475c8fbef239de91c279, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=127, earliestPutTs=1680062179153 2023-03-29 03:56:21,363 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting bc6336b3978840f4ab53bb5d285b6b7d, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=140, earliestPutTs=1680062179171 2023-03-29 03:56:21,363 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting f2ca8378cb984ac18483d4740747566b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=160, earliestPutTs=1680062179799 2023-03-29 03:56:21,364 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 67394449c9d94d699b3209ea6ff531ce, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=172, earliestPutTs=1680062180421 2023-03-29 03:56:21,372 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] throttle.PressureAwareThroughputController(145): 94b16ca160750195aa137348cdb80ba9#C#compaction#605 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:56:21,382 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/C/89d89c0b2327423b9a082530d2a61350 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/89d89c0b2327423b9a082530d2a61350 2023-03-29 03:56:21,386 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 94b16ca160750195aa137348cdb80ba9/C of 94b16ca160750195aa137348cdb80ba9 into 89d89c0b2327423b9a082530d2a61350(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:56:21,386 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 94b16ca160750195aa137348cdb80ba9: 2023-03-29 03:56:21,386 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9., storeName=94b16ca160750195aa137348cdb80ba9/C, priority=11, startTime=1680062181300; duration=0sec 2023-03-29 03:56:21,386 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:56:21,402 DEBUG [Thread-4303] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:56:21,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:56:21,404 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:56:21,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:56:21,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:56:21,405 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:56:21,405 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:21,405 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:56:21,405 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:56:21,405 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:21,406 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:21,407 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:21,407 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:21,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:21,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:21,407 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:21,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:21,407 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:56:21,407 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:21,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:56:21,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:21,408 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:56:21,408 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:56:21,408 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:56:21,408 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:56:21,408 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:21,408 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:56:21,412 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:56:21,412 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:56:21,412 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:21,412 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. started... 2023-03-29 03:56:21,412 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 94b16ca160750195aa137348cdb80ba9 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-29 03:56:21,412 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94b16ca160750195aa137348cdb80ba9, store=A 2023-03-29 03:56:21,412 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:56:21,412 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94b16ca160750195aa137348cdb80ba9, store=B 2023-03-29 03:56:21,412 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:56:21,413 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94b16ca160750195aa137348cdb80ba9, store=C 2023-03-29 03:56:21,413 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:56:21,426 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230329fc7b33d62b3e47f3a2771625e7c3801b to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230329fc7b33d62b3e47f3a2771625e7c3801b 2023-03-29 03:56:21,438 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=194, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/A/5d565495fa2048cfb38c6fcb5a7afb0d 2023-03-29 03:56:21,449 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=194 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/B/aa0f4a9b2728441c89a9c8fe10c5efc7 2023-03-29 03:56:21,461 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=194 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/C/f9497ac0271648b88593fd5c961819b6 2023-03-29 03:56:21,465 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/A/5d565495fa2048cfb38c6fcb5a7afb0d as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/5d565495fa2048cfb38c6fcb5a7afb0d 2023-03-29 03:56:21,469 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/5d565495fa2048cfb38c6fcb5a7afb0d, entries=150, sequenceid=194, filesize=25.3 K 2023-03-29 03:56:21,470 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/B/aa0f4a9b2728441c89a9c8fe10c5efc7 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/aa0f4a9b2728441c89a9c8fe10c5efc7 2023-03-29 03:56:21,473 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/aa0f4a9b2728441c89a9c8fe10c5efc7, entries=150, sequenceid=194, filesize=11.8 K 2023-03-29 03:56:21,473 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/C/f9497ac0271648b88593fd5c961819b6 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/f9497ac0271648b88593fd5c961819b6 2023-03-29 03:56:21,479 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/f9497ac0271648b88593fd5c961819b6, entries=150, sequenceid=194, filesize=11.8 K 2023-03-29 03:56:21,480 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=0 B/0 for 94b16ca160750195aa137348cdb80ba9 in 68ms, sequenceid=194, compaction requested=false 2023-03-29 03:56:21,480 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 94b16ca160750195aa137348cdb80ba9: 2023-03-29 03:56:21,480 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:21,480 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:56:21,480 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:56:21,480 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:21,480 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:56:21,480 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:56:21,482 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:21,482 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:21,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:21,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:21,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:21,482 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:21,482 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:56:21,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:21,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:21,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:21,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:21,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:21,484 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:56:21,484 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@552ab2e2[Count = 0] remaining members to acquire global barrier 2023-03-29 03:56:21,484 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:56:21,484 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:21,485 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:21,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:21,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:21,485 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:56:21,485 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:56:21,485 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:21,485 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:56:21,485 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:56:21,488 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:21,488 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:56:21,488 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:21,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:21,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:21,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:21,488 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:56:21,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:21,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:21,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:21,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:21,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:21,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:21,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:21,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:56:21,491 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:56:21,491 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:56:21,491 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:56:21,491 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:56:21,491 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:56:21,491 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:56:21,492 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:21,492 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:21,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:21,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:21,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:21,492 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:21,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:21,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:21,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:21,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:21,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:21,493 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:21,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:21,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:21,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:21,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:21,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:21,495 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:21,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:21,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:21,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:21,497 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:21,497 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:21,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:21,497 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:56:21,498 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:56:21,497 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:21,497 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:21,497 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:21,498 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:21,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:21,498 DEBUG [Thread-4303] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:56:21,498 DEBUG [Thread-4303] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:56:21,498 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:21,498 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:21,498 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:21,498 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:21,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:21,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:21,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on 94b16ca160750195aa137348cdb80ba9 2023-03-29 03:56:21,594 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 94b16ca160750195aa137348cdb80ba9 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-29 03:56:21,595 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94b16ca160750195aa137348cdb80ba9, store=A 2023-03-29 03:56:21,595 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:56:21,595 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94b16ca160750195aa137348cdb80ba9, store=B 2023-03-29 03:56:21,595 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:56:21,595 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94b16ca160750195aa137348cdb80ba9, store=C 2023-03-29 03:56:21,595 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:56:21,598 DEBUG [Thread-4303] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:56:21,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:56:21,600 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:56:21,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:56:21,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:56:21,604 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:56:21,604 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:21,604 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:56:21,604 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:56:21,605 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:21,605 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:21,606 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:21,606 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:21,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:21,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:21,607 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:21,607 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:56:21,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:21,607 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:21,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:56:21,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:21,608 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:56:21,608 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:56:21,608 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:56:21,608 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:21,608 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:56:21,612 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:56:21,612 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:56:21,612 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:21,612 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. started... 2023-03-29 03:56:21,613 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. as already flushing 2023-03-29 03:56:21,613 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:21,613 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:56:21,613 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:56:21,613 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:21,613 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:56:21,613 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:56:21,614 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680062181601, max timestamp=1680062181613Num uniques 859; 2023-03-29 03:56:21,614 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-29 03:56:21,615 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680062181601, max timestamp=1680062181613Num uniques 859; 2023-03-29 03:56:21,615 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-29 03:56:21,615 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680062181601, max timestamp=1680062181613Num uniques 859; 2023-03-29 03:56:21,615 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-29 03:56:21,615 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:21,615 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:21,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:21,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:21,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:21,616 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:21,616 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:56:21,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:21,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:21,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:21,617 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032990cbaa7c3659477f8336e27abcf91689 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032990cbaa7c3659477f8336e27abcf91689 2023-03-29 03:56:21,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:21,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:21,618 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:56:21,618 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3593ff54[Count = 0] remaining members to acquire global barrier 2023-03-29 03:56:21,618 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:56:21,619 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:21,620 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:21,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:21,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:21,621 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:56:21,621 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:56:21,621 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:56:21,621 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:21,621 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:56:21,622 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:56:21,623 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:21,623 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:21,623 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:56:21,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:21,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:21,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:21,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:21,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:21,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:21,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:21,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:21,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:21,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:21,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:56:21,628 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:56:21,628 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:56:21,628 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:56:21,628 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:56:21,628 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:56:21,628 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:56:21,630 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:21,630 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:21,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:21,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:21,630 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:21,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:21,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:21,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:21,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:21,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:21,630 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:21,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:21,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:21,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:21,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:21,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:21,632 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:21,632 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:21,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:21,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37674 deadline: 1680062241626, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:21,632 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:21,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37712 deadline: 1680062241628, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:21,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:21,632 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:21,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37652 deadline: 1680062241628, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:21,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:21,633 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:21,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37666 deadline: 1680062241629, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:21,635 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:21,635 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:21,635 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:21,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:21,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:21,635 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:21,635 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:21,635 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:21,635 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:56:21,635 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:56:21,635 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:21,635 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:21,636 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:21,636 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:21,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:21,636 DEBUG [Thread-4303] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:56:21,636 DEBUG [Thread-4303] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:56:21,635 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:56:21,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:21,637 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:21,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37670 deadline: 1680062241632, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:21,725 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/B/621c55fa87d34fbfaf88b0cd59718562 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/621c55fa87d34fbfaf88b0cd59718562 2023-03-29 03:56:21,735 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:21,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37674 deadline: 1680062241733, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:21,735 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:21,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37712 deadline: 1680062241733, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:21,736 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:21,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37652 deadline: 1680062241733, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:21,736 DEBUG [Thread-4303] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:56:21,736 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:21,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37666 deadline: 1680062241734, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:21,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:56:21,739 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 94b16ca160750195aa137348cdb80ba9/B of 94b16ca160750195aa137348cdb80ba9 into 621c55fa87d34fbfaf88b0cd59718562(size=11.9 K), total size for store is 23.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:56:21,739 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 94b16ca160750195aa137348cdb80ba9: 2023-03-29 03:56:21,739 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9., storeName=94b16ca160750195aa137348cdb80ba9/B, priority=11, startTime=1680062181300; duration=0sec 2023-03-29 03:56:21,739 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:56:21,740 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:56:21,740 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:21,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37670 deadline: 1680062241739, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:21,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:56:21,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:56:21,742 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:56:21,743 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:21,743 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:56:21,743 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:56:21,743 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:21,743 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:21,744 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:21,744 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:21,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:21,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:21,745 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:21,745 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:56:21,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:21,746 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:21,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:56:21,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:21,746 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:56:21,747 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:56:21,747 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:56:21,747 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:56:21,747 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:21,747 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:56:21,747 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:56:21,747 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:56:21,747 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:21,747 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. started... 2023-03-29 03:56:21,747 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. as already flushing 2023-03-29 03:56:21,747 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:21,747 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:56:21,748 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:56:21,748 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:21,748 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:56:21,748 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:56:21,749 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:21,749 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:21,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:21,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:21,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:21,750 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:21,750 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:56:21,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:21,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:21,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:21,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:21,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:21,752 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:56:21,752 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@590da9b9[Count = 0] remaining members to acquire global barrier 2023-03-29 03:56:21,752 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:56:21,752 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:21,758 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:21,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:21,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:21,759 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:56:21,759 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:56:21,759 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:21,759 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:56:21,759 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:56:21,760 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:56:21,760 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:21,760 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:21,760 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:56:21,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:21,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:21,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:21,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:21,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:21,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:21,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:21,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:21,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:21,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:21,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:56:21,763 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:56:21,763 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:56:21,763 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:56:21,763 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:56:21,763 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:56:21,763 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:56:21,765 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:21,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:21,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:21,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:21,765 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:21,765 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:21,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:21,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:21,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:21,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:21,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:21,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:21,766 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:21,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:21,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:21,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:21,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:21,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:21,767 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:21,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:21,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:21,770 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:21,770 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:21,770 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:21,770 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:56:21,770 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:56:21,770 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:21,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:21,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:21,770 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:21,770 DEBUG [Thread-4303] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:56:21,770 DEBUG [Thread-4303] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:56:21,770 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:21,770 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:21,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:21,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:21,770 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:21,771 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:21,771 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:21,791 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-03-29 03:56:21,870 DEBUG [Thread-4303] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:56:21,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:56:21,873 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:56:21,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:56:21,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:56:21,874 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:56:21,874 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:21,874 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:56:21,874 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:56:21,875 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:21,875 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:21,877 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:21,877 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:21,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:21,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:21,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:21,877 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:21,877 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:56:21,878 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:21,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:56:21,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:21,878 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:56:21,884 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:56:21,884 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:56:21,884 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:56:21,884 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:21,884 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:56:21,884 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:56:21,884 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:56:21,884 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:21,884 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. started... 2023-03-29 03:56:21,884 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. as already flushing 2023-03-29 03:56:21,884 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:21,884 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:56:21,885 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:56:21,885 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:21,885 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:56:21,885 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:56:21,886 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:21,886 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:21,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:21,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:21,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:21,887 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:21,887 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:56:21,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:21,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:21,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:21,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:21,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:21,888 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:56:21,888 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4910b674[Count = 0] remaining members to acquire global barrier 2023-03-29 03:56:21,888 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:56:21,888 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:21,890 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:21,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:21,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:21,890 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:56:21,890 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:56:21,890 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:56:21,890 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:21,890 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:56:21,891 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:21,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:21,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:21,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:21,892 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:56:21,892 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:21,892 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:56:21,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:21,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:21,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:21,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:21,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:21,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:21,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:21,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:56:21,894 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:56:21,894 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:56:21,894 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:56:21,894 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:56:21,894 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:56:21,894 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:56:21,897 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:21,897 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:21,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:21,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:21,897 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:21,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:21,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:21,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:21,898 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:21,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:21,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:21,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:21,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:21,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:21,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:21,899 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:21,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:21,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:21,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:21,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:21,902 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:21,902 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:21,902 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:56:21,902 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:56:21,902 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:21,902 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:56:21,902 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:21,902 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:21,902 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:21,902 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:21,902 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:21,902 DEBUG [Thread-4303] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:56:21,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:21,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:21,902 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:21,902 DEBUG [Thread-4303] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:56:21,902 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:21,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:21,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:21,938 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:21,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37712 deadline: 1680062241936, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:21,940 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:21,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37652 deadline: 1680062241937, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:21,940 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:21,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37674 deadline: 1680062241937, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:21,940 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:21,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37666 deadline: 1680062241938, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:21,945 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:21,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37670 deadline: 1680062241943, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:22,002 DEBUG [Thread-4303] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:56:22,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:56:22,004 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:56:22,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:56:22,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:56:22,005 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:56:22,005 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:22,006 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:56:22,006 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:56:22,006 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:22,006 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:22,007 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:22,007 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:22,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:22,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:22,008 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:22,008 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:56:22,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:22,008 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:22,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:56:22,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:22,008 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:56:22,009 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:56:22,009 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:56:22,009 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:56:22,009 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:22,009 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:56:22,009 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:56:22,009 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:22,009 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:56:22,009 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. started... 2023-03-29 03:56:22,009 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. as already flushing 2023-03-29 03:56:22,009 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:22,009 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:56:22,009 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:56:22,009 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:22,009 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:56:22,009 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:56:22,011 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:22,011 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:22,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:22,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:22,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:22,011 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:22,011 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:56:22,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:22,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:22,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:22,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:22,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:22,013 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:56:22,013 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@492da016[Count = 0] remaining members to acquire global barrier 2023-03-29 03:56:22,013 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:56:22,013 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:22,015 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:22,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:22,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:22,015 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:56:22,015 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:56:22,015 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:56:22,015 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:22,015 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:56:22,017 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:56:22,017 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:22,017 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:22,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:22,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:22,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:22,017 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:56:22,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:22,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:22,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:22,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:22,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:22,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:22,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:22,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:56:22,019 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:56:22,019 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:56:22,019 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:56:22,019 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:56:22,019 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:56:22,019 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:56:22,021 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:22,021 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:22,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:22,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:22,021 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:22,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:22,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:22,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:22,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:22,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:22,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:22,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:22,022 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:22,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:22,023 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:22,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:22,023 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=205, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/A/4b174ccdd8864f3e9f5577c828034cdf 2023-03-29 03:56:22,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:22,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:22,025 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:22,025 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:22,025 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:22,025 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:22,025 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:22,025 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:56:22,025 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:56:22,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:22,026 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:22,026 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:22,025 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:22,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:22,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:22,026 DEBUG [Thread-4303] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:56:22,025 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:22,026 DEBUG [Thread-4303] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:56:22,026 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:22,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:22,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:22,028 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:56:22,126 DEBUG [Thread-4303] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:56:22,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:56:22,128 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:56:22,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:56:22,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:56:22,130 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:56:22,130 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:22,130 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:56:22,130 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:56:22,130 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:22,130 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:22,132 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:22,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:22,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:22,132 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:22,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:22,133 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:22,133 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:56:22,133 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:22,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:56:22,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:22,134 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:56:22,134 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:56:22,134 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:56:22,134 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:56:22,134 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:22,134 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:56:22,134 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:56:22,134 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:56:22,134 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:22,134 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. started... 2023-03-29 03:56:22,134 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. as already flushing 2023-03-29 03:56:22,134 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:22,134 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:56:22,134 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:56:22,135 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:22,135 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:56:22,135 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:56:22,136 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:22,136 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:22,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:22,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:22,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:22,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:22,137 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:22,137 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:56:22,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:22,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:22,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:22,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:22,138 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:56:22,139 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@636e8f38[Count = 0] remaining members to acquire global barrier 2023-03-29 03:56:22,139 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:56:22,139 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:22,140 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:22,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:22,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:22,140 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:22,140 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:56:22,140 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:56:22,140 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:56:22,140 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:56:22,145 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:22,145 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:56:22,145 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:22,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:22,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:22,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:22,146 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:56:22,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:22,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:22,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:22,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:22,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:22,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:22,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:22,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:56:22,148 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:56:22,148 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:56:22,148 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:56:22,148 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:56:22,148 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:56:22,148 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:56:22,149 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:22,149 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:22,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:22,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:22,150 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:22,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:22,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:22,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:22,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:22,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:22,150 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:22,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:22,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:22,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:22,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:22,151 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:22,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:22,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:22,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:22,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:22,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:22,154 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:22,154 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:22,154 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:22,154 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:22,154 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:22,154 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:56:22,154 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:56:22,154 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:22,154 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:22,154 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:22,154 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:22,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:22,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:22,154 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:22,154 DEBUG [Thread-4303] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:56:22,154 DEBUG [Thread-4303] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:56:22,154 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:56:22,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:22,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:22,241 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:22,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37712 deadline: 1680062242240, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:22,242 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:22,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37652 deadline: 1680062242241, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:22,244 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:22,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37674 deadline: 1680062242243, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:22,245 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:22,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37666 deadline: 1680062242243, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:22,250 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:22,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37670 deadline: 1680062242248, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:22,255 DEBUG [Thread-4303] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:56:22,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:56:22,257 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:56:22,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:56:22,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:56:22,258 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:56:22,258 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:22,258 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:56:22,258 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:56:22,258 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:22,258 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:22,260 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:22,260 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:22,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:22,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:22,260 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:22,260 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:56:22,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:22,261 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:22,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:56:22,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:22,261 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:56:22,261 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:56:22,261 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:56:22,261 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:56:22,261 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:22,261 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:56:22,261 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:56:22,261 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:56:22,262 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:22,262 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. started... 2023-03-29 03:56:22,262 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. as already flushing 2023-03-29 03:56:22,262 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:22,262 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:56:22,262 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:56:22,262 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:22,262 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:56:22,262 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:56:22,265 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:22,265 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:22,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:22,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:22,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:22,265 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:22,265 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:56:22,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:22,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:22,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:22,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:22,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:22,267 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:56:22,267 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@579ab28d[Count = 0] remaining members to acquire global barrier 2023-03-29 03:56:22,267 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:56:22,267 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:22,268 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:22,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:22,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:22,268 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:56:22,268 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:56:22,268 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:56:22,268 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:22,268 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:56:22,270 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:22,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:22,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:22,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:22,270 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:56:22,270 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:22,270 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:56:22,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:22,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:22,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:22,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:22,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:22,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:22,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:22,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:56:22,273 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:56:22,273 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:56:22,273 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:56:22,273 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:56:22,273 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:56:22,273 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:56:22,274 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:22,275 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:22,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:22,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:22,275 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:22,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:22,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:22,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:22,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:22,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:22,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:22,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:22,276 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:22,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:22,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:22,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:22,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:22,277 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:22,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:22,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:22,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:22,281 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:22,281 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:22,281 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:22,281 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:22,281 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:22,281 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:22,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:22,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:22,281 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:22,281 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:56:22,281 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:56:22,281 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:22,282 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:22,282 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:22,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:22,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:22,282 DEBUG [Thread-4303] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:56:22,282 DEBUG [Thread-4303] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:56:22,382 DEBUG [Thread-4303] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:56:22,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:56:22,386 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:56:22,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:56:22,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:56:22,387 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:56:22,387 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:22,387 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:56:22,387 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:56:22,387 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:22,387 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:22,389 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:22,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:22,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:22,389 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:22,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:22,389 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:22,389 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:56:22,389 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:22,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:56:22,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:22,390 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:56:22,390 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:56:22,390 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:56:22,390 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:56:22,390 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:22,390 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:56:22,390 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:56:22,391 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:56:22,391 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:22,391 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. started... 2023-03-29 03:56:22,391 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. as already flushing 2023-03-29 03:56:22,391 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:22,391 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:56:22,391 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:56:22,391 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:22,391 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:56:22,391 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:56:22,392 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:22,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:22,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:22,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:22,393 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:22,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:22,393 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:22,393 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:56:22,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:22,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:22,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:22,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:22,394 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:56:22,394 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1bf50d1c[Count = 0] remaining members to acquire global barrier 2023-03-29 03:56:22,394 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:56:22,394 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:22,396 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:22,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:22,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:22,396 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:56:22,396 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:56:22,396 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:56:22,396 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:22,396 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:56:22,398 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:22,398 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:56:22,398 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:22,398 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:56:22,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:22,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:22,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:22,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:22,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:22,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:22,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:22,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:22,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:22,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:22,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:56:22,401 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:56:22,401 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:56:22,401 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:56:22,401 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:56:22,401 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:56:22,401 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:56:22,402 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:22,402 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:22,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:22,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:22,403 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:22,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:22,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:22,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:22,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:22,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:22,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:22,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:22,403 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:22,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:22,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:22,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:22,404 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:22,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:22,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:22,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:22,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:22,407 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:22,407 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:22,407 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:22,407 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:22,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:22,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:22,407 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:22,407 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:22,407 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:22,407 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:22,407 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:22,407 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:56:22,407 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:56:22,407 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:22,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:22,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:22,408 DEBUG [Thread-4303] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:56:22,408 DEBUG [Thread-4303] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:56:22,438 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=205 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/B/59b2356b806a45a58edc24610ac15efd 2023-03-29 03:56:22,448 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=205 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/C/52aaa9b60345448fb005082290b00994 2023-03-29 03:56:22,451 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/A/4b174ccdd8864f3e9f5577c828034cdf as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/4b174ccdd8864f3e9f5577c828034cdf 2023-03-29 03:56:22,455 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/4b174ccdd8864f3e9f5577c828034cdf, entries=400, sequenceid=205, filesize=59.4 K 2023-03-29 03:56:22,455 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/B/59b2356b806a45a58edc24610ac15efd as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/59b2356b806a45a58edc24610ac15efd 2023-03-29 03:56:22,458 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/59b2356b806a45a58edc24610ac15efd, entries=150, sequenceid=205, filesize=11.8 K 2023-03-29 03:56:22,459 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/C/52aaa9b60345448fb005082290b00994 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/52aaa9b60345448fb005082290b00994 2023-03-29 03:56:22,462 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/52aaa9b60345448fb005082290b00994, entries=150, sequenceid=205, filesize=11.8 K 2023-03-29 03:56:22,463 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 94b16ca160750195aa137348cdb80ba9 in 869ms, sequenceid=205, compaction requested=true 2023-03-29 03:56:22,463 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 94b16ca160750195aa137348cdb80ba9: 2023-03-29 03:56:22,463 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:56:22,463 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-29 03:56:22,463 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-29 03:56:22,463 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:56:22,463 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:56:22,464 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 112818 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-29 03:56:22,464 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36280 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-29 03:56:22,464 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1912): 94b16ca160750195aa137348cdb80ba9/B is initiating minor compaction (all files) 2023-03-29 03:56:22,464 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1912): 94b16ca160750195aa137348cdb80ba9/A is initiating minor compaction (all files) 2023-03-29 03:56:22,464 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 94b16ca160750195aa137348cdb80ba9/B in TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:22,464 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 94b16ca160750195aa137348cdb80ba9/A in TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:22,465 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/621c55fa87d34fbfaf88b0cd59718562, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/aa0f4a9b2728441c89a9c8fe10c5efc7, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/59b2356b806a45a58edc24610ac15efd] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp, totalSize=35.4 K 2023-03-29 03:56:22,465 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/2c7bb4c01e1346e7b210bbc1a227a40f, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/5d565495fa2048cfb38c6fcb5a7afb0d, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/4b174ccdd8864f3e9f5577c828034cdf] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp, totalSize=110.2 K 2023-03-29 03:56:22,465 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 621c55fa87d34fbfaf88b0cd59718562, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=172, earliestPutTs=1680062180421 2023-03-29 03:56:22,465 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 2c7bb4c01e1346e7b210bbc1a227a40f, keycount=150, bloomtype=ROW, size=25.5 K, encoding=NONE, compression=NONE, seqNum=172, earliestPutTs=1680062180421 2023-03-29 03:56:22,465 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting aa0f4a9b2728441c89a9c8fe10c5efc7, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=194, earliestPutTs=1680062180441 2023-03-29 03:56:22,466 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 5d565495fa2048cfb38c6fcb5a7afb0d, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=194, earliestPutTs=1680062180441 2023-03-29 03:56:22,466 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 59b2356b806a45a58edc24610ac15efd, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=205, earliestPutTs=1680062181589 2023-03-29 03:56:22,467 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 4b174ccdd8864f3e9f5577c828034cdf, keycount=400, bloomtype=ROW, size=59.4 K, encoding=NONE, compression=NONE, seqNum=205, earliestPutTs=1680062181580 2023-03-29 03:56:22,474 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] throttle.PressureAwareThroughputController(145): 94b16ca160750195aa137348cdb80ba9#B#compaction#612 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:56:22,478 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] throttle.PressureAwareThroughputController(145): 94b16ca160750195aa137348cdb80ba9#A#compaction#613 average throughput is 3.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:56:22,508 DEBUG [Thread-4303] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:56:22,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:56:22,511 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:56:22,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:56:22,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:56:22,513 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:56:22,514 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:22,514 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:56:22,514 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:56:22,514 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:22,514 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:22,515 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:22,515 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:22,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:22,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:22,516 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:22,516 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:56:22,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:22,516 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:22,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:56:22,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:22,517 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:56:22,517 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:56:22,517 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:56:22,517 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:56:22,517 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:22,517 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:56:22,517 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:56:22,517 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:56:22,517 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:22,517 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. started... 2023-03-29 03:56:22,518 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 94b16ca160750195aa137348cdb80ba9 3/3 column families, dataSize=127.47 KB heapSize=355.69 KB 2023-03-29 03:56:22,518 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94b16ca160750195aa137348cdb80ba9, store=A 2023-03-29 03:56:22,518 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-29 03:56:22,518 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94b16ca160750195aa137348cdb80ba9, store=B 2023-03-29 03:56:22,518 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-29 03:56:22,518 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94b16ca160750195aa137348cdb80ba9, store=C 2023-03-29 03:56:22,518 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-29 03:56:22,536 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303291f907ff0f69741b5a456b6c8794aa658 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303291f907ff0f69741b5a456b6c8794aa658 2023-03-29 03:56:22,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on 94b16ca160750195aa137348cdb80ba9 2023-03-29 03:56:22,746 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. as already flushing 2023-03-29 03:56:22,756 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:22,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37674 deadline: 1680062242751, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:22,757 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:22,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37652 deadline: 1680062242754, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:22,757 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:22,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37666 deadline: 1680062242755, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:22,760 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:22,761 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:22,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37670 deadline: 1680062242756, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:22,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37712 deadline: 1680062242756, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:22,860 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:22,860 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:22,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37674 deadline: 1680062242857, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:22,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37652 deadline: 1680062242858, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:22,861 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:22,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37666 deadline: 1680062242858, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:22,865 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:22,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37670 deadline: 1680062242861, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:22,866 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:22,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37712 deadline: 1680062242862, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:22,887 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/B/1676009485bb4324b35181b980cedc7d as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/1676009485bb4324b35181b980cedc7d 2023-03-29 03:56:22,891 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 94b16ca160750195aa137348cdb80ba9/B of 94b16ca160750195aa137348cdb80ba9 into 1676009485bb4324b35181b980cedc7d(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:56:22,891 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 94b16ca160750195aa137348cdb80ba9: 2023-03-29 03:56:22,891 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9., storeName=94b16ca160750195aa137348cdb80ba9/B, priority=13, startTime=1680062182463; duration=0sec 2023-03-29 03:56:22,891 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:56:22,891 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-29 03:56:22,892 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36280 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-29 03:56:22,893 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1912): 94b16ca160750195aa137348cdb80ba9/C is initiating minor compaction (all files) 2023-03-29 03:56:22,893 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 94b16ca160750195aa137348cdb80ba9/C in TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:22,893 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/89d89c0b2327423b9a082530d2a61350, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/f9497ac0271648b88593fd5c961819b6, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/52aaa9b60345448fb005082290b00994] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp, totalSize=35.4 K 2023-03-29 03:56:22,893 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 89d89c0b2327423b9a082530d2a61350, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=172, earliestPutTs=1680062180421 2023-03-29 03:56:22,893 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting f9497ac0271648b88593fd5c961819b6, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=194, earliestPutTs=1680062180441 2023-03-29 03:56:22,894 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 52aaa9b60345448fb005082290b00994, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=205, earliestPutTs=1680062181589 2023-03-29 03:56:22,898 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/A/54df16ea5a224f04869e997b7003f9fb as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/54df16ea5a224f04869e997b7003f9fb 2023-03-29 03:56:22,900 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] throttle.PressureAwareThroughputController(145): 94b16ca160750195aa137348cdb80ba9#C#compaction#615 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:56:22,902 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 94b16ca160750195aa137348cdb80ba9/A of 94b16ca160750195aa137348cdb80ba9 into 54df16ea5a224f04869e997b7003f9fb(size=25.6 K), total size for store is 25.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:56:22,902 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 94b16ca160750195aa137348cdb80ba9: 2023-03-29 03:56:22,902 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9., storeName=94b16ca160750195aa137348cdb80ba9/A, priority=13, startTime=1680062182463; duration=0sec 2023-03-29 03:56:22,902 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:56:22,907 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/C/e72e7307fb044791a44923488323e1d7 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/e72e7307fb044791a44923488323e1d7 2023-03-29 03:56:22,911 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 94b16ca160750195aa137348cdb80ba9/C of 94b16ca160750195aa137348cdb80ba9 into e72e7307fb044791a44923488323e1d7(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:56:22,911 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 94b16ca160750195aa137348cdb80ba9: 2023-03-29 03:56:22,911 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9., storeName=94b16ca160750195aa137348cdb80ba9/C, priority=13, startTime=1680062182463; duration=0sec 2023-03-29 03:56:22,911 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:56:22,940 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=240, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/A/84cb900ce4b94ae189ed1e7099c0124e 2023-03-29 03:56:22,958 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=240 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/B/84a4345f266c4fa599d5112825a964d3 2023-03-29 03:56:22,978 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=240 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/C/fcbdadeb4cc34a7fbe2ca17050241f80 2023-03-29 03:56:22,986 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/A/84cb900ce4b94ae189ed1e7099c0124e as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/84cb900ce4b94ae189ed1e7099c0124e 2023-03-29 03:56:22,990 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/84cb900ce4b94ae189ed1e7099c0124e, entries=150, sequenceid=240, filesize=25.3 K 2023-03-29 03:56:22,990 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/B/84a4345f266c4fa599d5112825a964d3 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/84a4345f266c4fa599d5112825a964d3 2023-03-29 03:56:22,994 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/84a4345f266c4fa599d5112825a964d3, entries=150, sequenceid=240, filesize=11.8 K 2023-03-29 03:56:22,995 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/C/fcbdadeb4cc34a7fbe2ca17050241f80 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/fcbdadeb4cc34a7fbe2ca17050241f80 2023-03-29 03:56:22,998 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/fcbdadeb4cc34a7fbe2ca17050241f80, entries=150, sequenceid=240, filesize=11.8 K 2023-03-29 03:56:22,998 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~355.64 KB/364176, currentSize=53.67 KB/54960 for 94b16ca160750195aa137348cdb80ba9 in 481ms, sequenceid=240, compaction requested=false 2023-03-29 03:56:22,999 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 94b16ca160750195aa137348cdb80ba9: 2023-03-29 03:56:22,999 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:22,999 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:56:22,999 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:56:22,999 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:22,999 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:56:22,999 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:56:23,001 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,001 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:23,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:23,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:23,002 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:23,002 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:56:23,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:23,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:23,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:23,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:23,004 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:56:23,004 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3f7c2364[Count = 0] remaining members to acquire global barrier 2023-03-29 03:56:23,004 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:56:23,004 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:23,006 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:23,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:23,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:23,006 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:56:23,006 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:56:23,006 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:56:23,006 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,006 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:56:23,008 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:56:23,008 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,008 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:23,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:23,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:23,008 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:56:23,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:23,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:23,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:23,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:23,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:23,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:56:23,011 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:56:23,011 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:56:23,011 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:56:23,011 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:56:23,011 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:56:23,011 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:56:23,013 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:23,013 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:23,013 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:23,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:23,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:23,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:23,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:23,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:23,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:23,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:23,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:23,014 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:23,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:23,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:23,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:23,015 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:23,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:23,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,023 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,023 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:23,023 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:23,023 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:23,023 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:56:23,023 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:56:23,023 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:23,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:23,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:23,023 DEBUG [Thread-4303] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:56:23,023 DEBUG [Thread-4303] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:56:23,023 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,023 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:23,023 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:23,023 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:23,023 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:23,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:23,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:23,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on 94b16ca160750195aa137348cdb80ba9 2023-03-29 03:56:23,063 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 94b16ca160750195aa137348cdb80ba9 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-29 03:56:23,065 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94b16ca160750195aa137348cdb80ba9, store=A 2023-03-29 03:56:23,065 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:56:23,065 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94b16ca160750195aa137348cdb80ba9, store=B 2023-03-29 03:56:23,065 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:56:23,065 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94b16ca160750195aa137348cdb80ba9, store=C 2023-03-29 03:56:23,065 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:56:23,079 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:23,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37712 deadline: 1680062243077, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,080 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:23,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37652 deadline: 1680062243077, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,080 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:23,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37666 deadline: 1680062243077, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,080 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:23,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37674 deadline: 1680062243078, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,081 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:23,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37670 deadline: 1680062243079, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,123 DEBUG [Thread-4303] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:56:23,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:56:23,125 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:56:23,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:56:23,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:56:23,127 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:56:23,127 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:23,127 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:56:23,127 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:56:23,127 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:23,127 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:23,131 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:23,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:23,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:23,131 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:23,132 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,132 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:56:23,132 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:23,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:56:23,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:23,132 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:56:23,132 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:56:23,132 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:56:23,132 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:56:23,133 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:23,133 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:56:23,133 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:56:23,133 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:56:23,133 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:23,133 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. started... 2023-03-29 03:56:23,133 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. as already flushing 2023-03-29 03:56:23,133 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:23,133 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:56:23,133 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:56:23,133 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,133 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:56:23,133 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:56:23,135 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,135 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:23,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:23,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:23,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:23,135 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:23,136 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:56:23,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:23,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:23,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:23,137 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:56:23,138 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@28fc7cfc[Count = 0] remaining members to acquire global barrier 2023-03-29 03:56:23,138 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:56:23,138 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:23,139 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:23,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:23,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:23,139 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,139 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:56:23,139 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:56:23,139 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:56:23,139 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:56:23,142 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,142 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:56:23,142 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:23,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:23,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:23,142 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:56:23,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:23,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:23,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:23,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:23,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:23,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:56:23,145 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:56:23,145 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:56:23,145 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:56:23,145 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:56:23,145 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:56:23,145 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:56:23,147 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:23,147 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:23,147 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:23,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:23,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:23,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:23,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:23,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:23,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:23,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:23,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:23,148 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:23,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:23,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:23,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:23,149 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:23,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:23,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,153 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,153 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:23,153 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:23,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:23,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:23,153 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:23,153 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:23,153 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:23,153 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:56:23,153 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:56:23,153 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,153 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:23,153 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:23,153 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:23,153 DEBUG [Thread-4303] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:56:23,154 DEBUG [Thread-4303] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:56:23,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:23,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:23,183 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:23,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37712 deadline: 1680062243181, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,183 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:23,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37652 deadline: 1680062243181, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,183 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:23,184 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:23,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37666 deadline: 1680062243181, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37674 deadline: 1680062243181, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,184 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:23,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37670 deadline: 1680062243182, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,254 DEBUG [Thread-4303] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:56:23,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:56:23,256 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:56:23,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:56:23,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:56:23,257 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:56:23,257 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:23,257 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:56:23,257 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:56:23,264 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:23,264 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:23,266 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,266 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:23,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:23,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:23,266 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,266 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:56:23,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:23,267 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:23,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:56:23,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:23,267 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:56:23,268 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:56:23,268 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:56:23,268 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:56:23,268 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:23,268 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:56:23,268 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:56:23,268 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:56:23,268 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:23,268 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. started... 2023-03-29 03:56:23,268 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. as already flushing 2023-03-29 03:56:23,268 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:23,268 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:56:23,268 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:56:23,268 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,268 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:56:23,268 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:56:23,271 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,271 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:23,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:23,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:23,272 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:23,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:23,272 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:56:23,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:23,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:23,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:23,274 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:56:23,274 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2a70bdb4[Count = 0] remaining members to acquire global barrier 2023-03-29 03:56:23,274 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:56:23,274 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:23,275 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:23,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:23,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:23,275 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:56:23,275 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:56:23,275 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:56:23,276 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,276 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:56:23,277 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,277 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:56:23,278 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:23,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:23,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:23,278 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:56:23,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:23,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:23,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:23,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:23,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:23,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:56:23,280 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:56:23,280 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:56:23,280 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:56:23,280 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:56:23,280 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:56:23,280 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:56:23,282 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:23,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:23,282 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:23,282 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:23,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:23,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:23,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:23,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:23,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:23,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:23,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:23,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:23,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:23,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:23,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,284 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:23,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:23,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:23,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,285 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,288 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,288 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:23,288 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:23,288 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:23,288 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:23,288 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:23,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:23,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:23,288 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:56:23,289 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:56:23,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:23,288 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:23,289 DEBUG [Thread-4303] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:56:23,289 DEBUG [Thread-4303] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:56:23,289 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:23,289 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:23,289 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:23,386 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:23,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37674 deadline: 1680062243385, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,386 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:23,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37712 deadline: 1680062243385, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,386 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:23,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37652 deadline: 1680062243385, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,387 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:23,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37666 deadline: 1680062243386, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,387 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:23,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37670 deadline: 1680062243386, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,389 DEBUG [Thread-4303] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:56:23,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:56:23,391 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:56:23,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:56:23,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:56:23,392 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:56:23,392 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:23,393 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:56:23,393 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:56:23,393 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:23,393 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:23,394 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:23,394 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:23,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:23,395 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,395 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:56:23,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:23,395 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:23,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:56:23,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:23,396 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:56:23,396 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:56:23,396 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:56:23,396 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:56:23,396 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:23,396 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:56:23,396 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:56:23,396 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:56:23,396 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:23,396 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. started... 2023-03-29 03:56:23,396 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. as already flushing 2023-03-29 03:56:23,396 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:23,397 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:56:23,397 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:56:23,397 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,397 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:56:23,397 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:56:23,398 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,398 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:23,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:23,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:23,399 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:23,399 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:56:23,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:23,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:23,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:23,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:23,400 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:56:23,400 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@19dec80b[Count = 0] remaining members to acquire global barrier 2023-03-29 03:56:23,400 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:56:23,400 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:23,401 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:23,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:23,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:23,402 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:56:23,402 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:56:23,402 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:56:23,402 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,402 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:56:23,404 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:56:23,404 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:23,404 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,404 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:56:23,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:23,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:23,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:23,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:23,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:23,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:23,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:23,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:56:23,407 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:56:23,407 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:56:23,407 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:56:23,407 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:56:23,407 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:56:23,407 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:56:23,409 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:23,409 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:23,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:23,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:23,409 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,409 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:23,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:23,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:23,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:23,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:23,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:23,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:23,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:23,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:23,410 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:23,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:23,412 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:23,412 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:23,413 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:56:23,413 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:56:23,413 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:23,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:23,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:23,413 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:23,413 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:23,413 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:23,413 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,413 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:23,413 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:23,413 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:23,413 DEBUG [Thread-4303] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:56:23,413 DEBUG [Thread-4303] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:56:23,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:23,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:23,414 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:56:23,477 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230329ba149981e56348e2ab0378ef72dd1f24 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230329ba149981e56348e2ab0378ef72dd1f24 2023-03-29 03:56:23,513 DEBUG [Thread-4303] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:56:23,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:56:23,515 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:56:23,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:56:23,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:56:23,517 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:56:23,517 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:23,517 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:56:23,517 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:56:23,517 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:23,517 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:23,520 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,520 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:23,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:23,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:23,520 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,520 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:56:23,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:23,520 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:23,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:56:23,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:23,521 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:56:23,521 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:56:23,521 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:56:23,521 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:56:23,521 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:23,521 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:56:23,522 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:56:23,522 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:56:23,522 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:23,522 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. started... 2023-03-29 03:56:23,522 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. as already flushing 2023-03-29 03:56:23,522 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:23,522 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:56:23,522 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:56:23,522 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,522 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:56:23,522 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:56:23,524 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,524 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:23,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:23,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:23,524 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:23,524 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:56:23,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:23,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:23,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:23,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:23,526 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:56:23,526 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4767af57[Count = 0] remaining members to acquire global barrier 2023-03-29 03:56:23,526 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:56:23,526 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:23,528 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:23,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:23,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:23,528 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:56:23,528 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:56:23,528 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:56:23,528 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,528 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:56:23,530 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,530 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:56:23,530 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:23,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,530 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:56:23,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:23,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:23,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:23,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:23,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:23,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:23,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:23,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:56:23,532 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:56:23,532 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:56:23,532 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:56:23,532 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:56:23,532 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:56:23,532 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:56:23,535 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:23,535 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:23,535 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:23,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:23,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:23,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:23,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:23,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:23,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:23,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:23,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:23,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:23,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:23,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:23,536 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:23,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:23,537 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:23,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,540 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,540 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:23,540 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:23,540 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,540 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:23,540 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:23,540 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:56:23,540 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:56:23,540 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:23,540 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:23,540 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:23,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:23,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:23,540 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:23,540 DEBUG [Thread-4303] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:56:23,540 DEBUG [Thread-4303] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:56:23,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:23,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:23,640 DEBUG [Thread-4303] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:56:23,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:56:23,642 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:56:23,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:56:23,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:56:23,644 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:56:23,644 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:23,644 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:56:23,644 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:56:23,644 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:23,644 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:23,646 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:23,646 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:23,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:23,646 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,646 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:56:23,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:23,647 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:23,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:56:23,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:23,647 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:56:23,647 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:56:23,647 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:56:23,647 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:56:23,647 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:23,647 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:56:23,648 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:56:23,648 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:56:23,648 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:23,648 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. started... 2023-03-29 03:56:23,648 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. as already flushing 2023-03-29 03:56:23,648 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:23,648 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:56:23,648 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:56:23,648 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,648 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:56:23,648 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:56:23,649 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,650 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:23,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:23,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:23,650 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:23,650 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:56:23,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:23,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:23,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:23,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:23,651 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:56:23,651 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@c3730db[Count = 0] remaining members to acquire global barrier 2023-03-29 03:56:23,651 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:56:23,651 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:23,654 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:23,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:23,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:23,654 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,654 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:56:23,654 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:56:23,655 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:56:23,655 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:56:23,656 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,656 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:56:23,656 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:23,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:23,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:23,656 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:56:23,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:23,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:23,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:23,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:23,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:23,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:56:23,658 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:56:23,658 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:56:23,658 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:56:23,658 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:56:23,659 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:56:23,659 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:56:23,660 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:23,660 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:23,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:23,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:23,660 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:23,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:23,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:23,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:23,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:23,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:23,661 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:23,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:23,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:23,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:23,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:23,663 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:23,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:23,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,665 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,665 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:23,665 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:23,665 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,665 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:23,665 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:23,665 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:56:23,665 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:56:23,666 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:56:23,665 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:23,666 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:23,665 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:23,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:23,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:23,665 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:23,666 DEBUG [Thread-4303] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:56:23,666 DEBUG [Thread-4303] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:56:23,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:23,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:23,688 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:23,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37652 deadline: 1680062243687, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,688 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:23,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37666 deadline: 1680062243687, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,690 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:23,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37674 deadline: 1680062243688, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,690 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:23,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37712 deadline: 1680062243688, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,692 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:23,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37670 deadline: 1680062243690, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,766 DEBUG [Thread-4303] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:56:23,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:56:23,768 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:56:23,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:56:23,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:56:23,769 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:56:23,769 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:23,770 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:56:23,770 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:56:23,770 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:23,770 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:23,772 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:23,772 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:23,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:23,772 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,772 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:56:23,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:23,773 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:23,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:56:23,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:23,773 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:56:23,773 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:56:23,773 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:56:23,773 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:56:23,774 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:23,774 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:56:23,774 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:56:23,774 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:56:23,774 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:23,774 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. started... 2023-03-29 03:56:23,774 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. as already flushing 2023-03-29 03:56:23,774 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:23,774 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:56:23,774 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:56:23,774 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,774 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:56:23,774 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:56:23,776 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:23,776 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:23,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:23,776 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:23,776 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:56:23,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:23,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:23,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:23,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:23,777 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:56:23,777 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2305a1f4[Count = 0] remaining members to acquire global barrier 2023-03-29 03:56:23,777 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:56:23,778 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:23,779 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:23,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:23,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:23,779 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:56:23,779 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:56:23,779 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:56:23,779 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,779 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:56:23,780 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,780 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:56:23,781 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:23,781 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:56:23,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:23,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:23,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:23,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:23,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:23,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:23,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:23,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:56:23,783 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:56:23,783 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:56:23,783 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:56:23,783 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:56:23,783 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:56:23,783 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:56:23,786 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:23,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:23,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:23,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:23,786 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:23,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:23,786 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:23,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:23,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:23,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:23,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:23,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:23,787 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:23,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:23,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:23,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:23,788 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:23,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,791 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:23,791 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:23,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:23,791 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:56:23,791 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:56:23,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:23,791 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:23,791 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,791 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:23,791 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:23,791 DEBUG [Thread-4303] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:56:23,791 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:23,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:23,791 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:23,791 DEBUG [Thread-4303] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:56:23,792 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:23,792 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:23,883 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=257, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/A/629361042e7d4e15996be65d1b80128c 2023-03-29 03:56:23,892 DEBUG [Thread-4303] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:56:23,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:56:23,894 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:56:23,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:56:23,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:56:23,896 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:56:23,896 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:23,896 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:56:23,896 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:56:23,896 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:23,896 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:23,899 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:23,900 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:23,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:23,900 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,900 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:56:23,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:23,900 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:23,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:56:23,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:23,901 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:56:23,901 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:56:23,901 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:56:23,901 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:56:23,901 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:23,901 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:56:23,902 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:56:23,902 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:56:23,902 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:23,902 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. started... 2023-03-29 03:56:23,902 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. as already flushing 2023-03-29 03:56:23,902 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:23,902 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:56:23,902 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:56:23,902 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,902 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:56:23,902 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:56:23,905 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,905 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:23,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:23,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:23,906 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:23,906 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:56:23,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:23,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:23,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:23,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:23,907 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:56:23,907 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c74ab0f[Count = 0] remaining members to acquire global barrier 2023-03-29 03:56:23,908 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:56:23,908 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:23,909 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:23,909 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:23,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:23,909 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:56:23,909 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:56:23,909 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:56:23,909 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,909 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:56:23,911 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,911 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:56:23,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:23,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:23,911 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:23,911 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:56:23,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:23,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:23,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:23,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:23,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:23,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:56:23,915 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:56:23,915 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:56:23,915 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:56:23,915 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:56:23,915 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:56:23,915 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:56:23,917 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:23,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:23,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:23,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:23,917 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:23,917 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:23,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:23,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:23,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:23,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:23,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:23,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:23,918 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:23,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:23,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:23,919 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:23,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:23,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,922 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,922 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:23,922 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:23,922 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:23,923 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:23,923 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:23,922 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:23,923 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:23,923 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:23,923 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:23,922 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:56:23,923 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:56:23,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:23,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:23,923 DEBUG [Thread-4303] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:56:23,923 DEBUG [Thread-4303] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:56:23,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:23,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:23,923 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:56:24,023 DEBUG [Thread-4303] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:56:24,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:56:24,026 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:56:24,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:56:24,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:56:24,027 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:56:24,027 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:24,027 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:56:24,027 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:56:24,027 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:24,027 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:24,029 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:24,029 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:24,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:24,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:24,029 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:24,029 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:56:24,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:24,029 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:24,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:56:24,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:24,030 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:56:24,030 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:56:24,030 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:56:24,030 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:56:24,030 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:24,030 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:56:24,031 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:56:24,031 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:56:24,031 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:24,031 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. started... 2023-03-29 03:56:24,031 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. as already flushing 2023-03-29 03:56:24,031 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:24,031 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:56:24,031 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:56:24,031 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:24,031 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:56:24,031 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:56:24,032 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:24,032 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:24,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:24,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:24,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:24,033 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:24,033 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:56:24,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:24,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:24,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:24,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:24,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:24,034 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:56:24,034 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3693b8c1[Count = 0] remaining members to acquire global barrier 2023-03-29 03:56:24,034 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:56:24,034 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:24,035 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:24,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:24,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:24,036 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:56:24,036 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:56:24,036 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:56:24,036 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:24,036 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:56:24,039 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:56:24,039 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:24,039 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:24,039 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:56:24,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:24,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:24,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:24,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:24,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:24,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:24,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:24,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:24,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:24,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:24,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:56:24,041 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:56:24,041 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:56:24,041 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:56:24,041 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:56:24,041 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:56:24,041 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:56:24,043 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:24,043 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:24,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:24,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:24,043 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:24,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:24,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:24,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:24,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:24,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:24,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:24,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:24,044 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:24,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:24,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:24,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:24,045 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:24,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:24,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:24,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:24,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:24,047 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:24,047 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:24,047 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:24,047 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:24,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:24,047 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:56:24,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:24,047 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:24,047 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:56:24,047 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:56:24,047 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:24,047 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:24,047 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:24,048 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:24,048 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:24,048 DEBUG [Thread-4303] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:56:24,048 DEBUG [Thread-4303] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:56:24,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:24,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:24,148 DEBUG [Thread-4303] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:56:24,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:56:24,150 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:56:24,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:56:24,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:56:24,151 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:56:24,151 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:24,151 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:56:24,151 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:56:24,151 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:24,151 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:24,152 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:24,153 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:24,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:24,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:24,153 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:24,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:24,153 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:56:24,153 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:24,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:56:24,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:24,154 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:56:24,154 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:56:24,154 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:56:24,154 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:56:24,154 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:24,154 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:56:24,154 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:56:24,154 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:56:24,154 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:24,154 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. started... 2023-03-29 03:56:24,154 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. as already flushing 2023-03-29 03:56:24,155 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:24,155 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:56:24,155 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:56:24,155 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:24,155 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:56:24,155 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:56:24,157 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:24,157 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:24,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:24,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:24,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:24,158 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:24,158 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:56:24,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:24,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:24,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:24,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:24,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:24,159 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:56:24,159 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@317e720[Count = 0] remaining members to acquire global barrier 2023-03-29 03:56:24,159 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:56:24,159 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:24,160 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:24,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:24,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:24,160 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:56:24,160 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:56:24,160 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:56:24,160 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:24,161 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:56:24,162 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:24,162 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:56:24,162 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:24,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:24,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:24,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:24,162 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:56:24,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:24,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:24,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:24,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:24,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:24,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:24,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:24,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:56:24,164 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:56:24,164 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:56:24,164 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:56:24,164 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:56:24,164 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:56:24,164 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:56:24,166 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:24,166 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:24,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:24,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:24,166 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:24,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:24,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:24,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:24,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:24,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:24,166 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:24,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:24,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:24,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:24,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:24,167 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:24,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:24,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:24,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:24,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:24,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:24,171 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:24,171 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:24,171 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:24,171 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:24,171 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:24,171 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:56:24,171 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:56:24,171 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:24,171 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:24,171 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:24,171 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:24,171 DEBUG [Thread-4303] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:56:24,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:24,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:24,171 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:56:24,171 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:24,171 DEBUG [Thread-4303] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:56:24,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:24,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:24,192 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:24,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37666 deadline: 1680062244189, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:24,193 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:24,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37652 deadline: 1680062244191, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:24,195 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:24,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37712 deadline: 1680062244193, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:24,195 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:24,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37674 deadline: 1680062244194, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:24,197 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:24,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37670 deadline: 1680062244196, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:24,272 DEBUG [Thread-4303] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:56:24,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:56:24,274 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:56:24,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:56:24,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:56:24,275 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:56:24,275 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:24,275 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:56:24,275 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:56:24,276 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:24,276 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:24,277 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:24,277 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:24,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:24,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:24,278 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:24,278 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:56:24,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:24,278 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:24,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:56:24,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:24,279 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:56:24,279 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:56:24,279 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:56:24,279 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:56:24,279 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:24,279 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:56:24,279 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:56:24,279 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:56:24,279 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:24,279 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. started... 2023-03-29 03:56:24,279 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. as already flushing 2023-03-29 03:56:24,280 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:24,280 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:56:24,280 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:56:24,280 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:24,280 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:56:24,280 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:56:24,281 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:24,281 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:24,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:24,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:24,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:24,282 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:24,282 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:56:24,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:24,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:24,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:24,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:24,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:24,283 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:56:24,283 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@d13ecb7[Count = 0] remaining members to acquire global barrier 2023-03-29 03:56:24,283 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:56:24,284 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:24,288 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:24,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:24,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:24,288 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:56:24,288 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:56:24,288 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:56:24,288 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:24,288 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:56:24,290 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:24,290 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:56:24,290 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:24,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:24,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:24,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:24,290 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:56:24,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:24,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:24,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:24,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:24,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:24,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:24,292 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=257 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/B/a9e314fab9b8438b800d7b027f45c676 2023-03-29 03:56:24,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:24,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:56:24,292 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:56:24,292 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:56:24,293 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:56:24,293 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:56:24,293 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:56:24,293 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:56:24,295 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:24,295 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:24,295 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:24,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:24,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:24,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:24,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:24,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:24,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:24,296 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:24,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:24,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:24,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:24,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:24,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:24,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:24,298 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:24,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:24,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:24,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:24,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:24,301 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:24,301 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:24,301 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:56:24,301 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:24,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:24,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:24,301 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:56:24,301 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:56:24,301 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:24,301 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:24,301 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:24,301 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:24,301 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:24,301 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:24,301 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:24,301 DEBUG [Thread-4303] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:56:24,301 DEBUG [Thread-4303] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:56:24,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:24,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:24,303 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=257 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/C/b0e1d19cd0614a2d942e5b1301fbceb0 2023-03-29 03:56:24,307 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/A/629361042e7d4e15996be65d1b80128c as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/629361042e7d4e15996be65d1b80128c 2023-03-29 03:56:24,310 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/629361042e7d4e15996be65d1b80128c, entries=200, sequenceid=257, filesize=32.2 K 2023-03-29 03:56:24,311 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/B/a9e314fab9b8438b800d7b027f45c676 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/a9e314fab9b8438b800d7b027f45c676 2023-03-29 03:56:24,314 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/a9e314fab9b8438b800d7b027f45c676, entries=150, sequenceid=257, filesize=11.8 K 2023-03-29 03:56:24,315 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/C/b0e1d19cd0614a2d942e5b1301fbceb0 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/b0e1d19cd0614a2d942e5b1301fbceb0 2023-03-29 03:56:24,318 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/b0e1d19cd0614a2d942e5b1301fbceb0, entries=150, sequenceid=257, filesize=11.8 K 2023-03-29 03:56:24,319 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 94b16ca160750195aa137348cdb80ba9 in 1256ms, sequenceid=257, compaction requested=true 2023-03-29 03:56:24,319 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 94b16ca160750195aa137348cdb80ba9: 2023-03-29 03:56:24,319 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:56:24,319 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-29 03:56:24,319 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-29 03:56:24,319 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:56:24,319 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:56:24,320 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 85002 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-29 03:56:24,320 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1912): 94b16ca160750195aa137348cdb80ba9/A is initiating minor compaction (all files) 2023-03-29 03:56:24,320 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36432 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-29 03:56:24,320 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 94b16ca160750195aa137348cdb80ba9/A in TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:24,320 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1912): 94b16ca160750195aa137348cdb80ba9/B is initiating minor compaction (all files) 2023-03-29 03:56:24,320 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/54df16ea5a224f04869e997b7003f9fb, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/84cb900ce4b94ae189ed1e7099c0124e, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/629361042e7d4e15996be65d1b80128c] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp, totalSize=83.0 K 2023-03-29 03:56:24,320 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 94b16ca160750195aa137348cdb80ba9/B in TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:24,320 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/1676009485bb4324b35181b980cedc7d, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/84a4345f266c4fa599d5112825a964d3, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/a9e314fab9b8438b800d7b027f45c676] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp, totalSize=35.6 K 2023-03-29 03:56:24,321 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 54df16ea5a224f04869e997b7003f9fb, keycount=150, bloomtype=ROW, size=25.6 K, encoding=NONE, compression=NONE, seqNum=205, earliestPutTs=1680062181589 2023-03-29 03:56:24,321 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 1676009485bb4324b35181b980cedc7d, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=205, earliestPutTs=1680062181589 2023-03-29 03:56:24,321 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 84cb900ce4b94ae189ed1e7099c0124e, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=240, earliestPutTs=1680062181625 2023-03-29 03:56:24,321 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 84a4345f266c4fa599d5112825a964d3, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=240, earliestPutTs=1680062181625 2023-03-29 03:56:24,321 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 629361042e7d4e15996be65d1b80128c, keycount=200, bloomtype=ROW, size=32.2 K, encoding=NONE, compression=NONE, seqNum=257, earliestPutTs=1680062182755 2023-03-29 03:56:24,322 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting a9e314fab9b8438b800d7b027f45c676, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=257, earliestPutTs=1680062182755 2023-03-29 03:56:24,327 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] throttle.PressureAwareThroughputController(145): 94b16ca160750195aa137348cdb80ba9#B#compaction#622 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:56:24,330 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] throttle.PressureAwareThroughputController(145): 94b16ca160750195aa137348cdb80ba9#A#compaction#621 average throughput is 4.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:56:24,402 DEBUG [Thread-4303] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:56:24,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:56:24,403 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:56:24,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:56:24,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:56:24,404 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:56:24,404 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:24,405 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:56:24,405 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:56:24,405 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:24,405 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:24,408 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:24,408 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:24,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:24,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:24,408 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:24,408 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:56:24,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:24,409 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:24,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:56:24,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:24,409 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:56:24,410 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:56:24,410 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:56:24,410 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:56:24,410 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:24,410 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:56:24,410 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:56:24,410 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:56:24,410 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:24,410 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. started... 2023-03-29 03:56:24,410 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 94b16ca160750195aa137348cdb80ba9 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-03-29 03:56:24,410 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94b16ca160750195aa137348cdb80ba9, store=A 2023-03-29 03:56:24,410 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:56:24,410 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94b16ca160750195aa137348cdb80ba9, store=B 2023-03-29 03:56:24,410 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:56:24,410 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94b16ca160750195aa137348cdb80ba9, store=C 2023-03-29 03:56:24,410 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:56:24,745 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/B/37682f23d96a415593111dc008377dcd as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/37682f23d96a415593111dc008377dcd 2023-03-29 03:56:24,749 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 94b16ca160750195aa137348cdb80ba9/B of 94b16ca160750195aa137348cdb80ba9 into 37682f23d96a415593111dc008377dcd(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:56:24,749 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 94b16ca160750195aa137348cdb80ba9: 2023-03-29 03:56:24,749 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9., storeName=94b16ca160750195aa137348cdb80ba9/B, priority=13, startTime=1680062184319; duration=0sec 2023-03-29 03:56:24,749 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:56:24,749 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-29 03:56:24,750 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36432 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-29 03:56:24,750 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1912): 94b16ca160750195aa137348cdb80ba9/C is initiating minor compaction (all files) 2023-03-29 03:56:24,750 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 94b16ca160750195aa137348cdb80ba9/C in TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:24,750 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/e72e7307fb044791a44923488323e1d7, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/fcbdadeb4cc34a7fbe2ca17050241f80, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/b0e1d19cd0614a2d942e5b1301fbceb0] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp, totalSize=35.6 K 2023-03-29 03:56:24,751 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting e72e7307fb044791a44923488323e1d7, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=205, earliestPutTs=1680062181589 2023-03-29 03:56:24,751 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting fcbdadeb4cc34a7fbe2ca17050241f80, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=240, earliestPutTs=1680062181625 2023-03-29 03:56:24,751 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting b0e1d19cd0614a2d942e5b1301fbceb0, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=257, earliestPutTs=1680062182755 2023-03-29 03:56:24,756 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] throttle.PressureAwareThroughputController(145): 94b16ca160750195aa137348cdb80ba9#C#compaction#624 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:56:24,769 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/A/f40864718506408c8d813380f86cead0 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/f40864718506408c8d813380f86cead0 2023-03-29 03:56:24,772 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 94b16ca160750195aa137348cdb80ba9/A of 94b16ca160750195aa137348cdb80ba9 into f40864718506408c8d813380f86cead0(size=25.7 K), total size for store is 25.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:56:24,772 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 94b16ca160750195aa137348cdb80ba9: 2023-03-29 03:56:24,772 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9., storeName=94b16ca160750195aa137348cdb80ba9/A, priority=13, startTime=1680062184319; duration=0sec 2023-03-29 03:56:24,773 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:56:24,827 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032999e975310e0f4eb08355fdecf72eeb98 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032999e975310e0f4eb08355fdecf72eeb98 2023-03-29 03:56:24,832 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=275, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/A/4bf75005213b4e6b88cafa073e2e8d13 2023-03-29 03:56:25,165 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/C/d284df99f2824152baff1fae57af725a as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/d284df99f2824152baff1fae57af725a 2023-03-29 03:56:25,168 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 94b16ca160750195aa137348cdb80ba9/C of 94b16ca160750195aa137348cdb80ba9 into d284df99f2824152baff1fae57af725a(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:56:25,169 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 94b16ca160750195aa137348cdb80ba9: 2023-03-29 03:56:25,169 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9., storeName=94b16ca160750195aa137348cdb80ba9/C, priority=13, startTime=1680062184319; duration=0sec 2023-03-29 03:56:25,169 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:56:25,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on 94b16ca160750195aa137348cdb80ba9 2023-03-29 03:56:25,196 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. as already flushing 2023-03-29 03:56:25,229 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:25,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37670 deadline: 1680062245226, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:25,230 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:25,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37652 deadline: 1680062245226, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:25,231 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:25,231 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:25,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37712 deadline: 1680062245228, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:25,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37674 deadline: 1680062245229, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:25,232 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:25,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37666 deadline: 1680062245230, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:25,243 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=275 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/B/32523715f3fb4f3694681eb8024aa11d 2023-03-29 03:56:25,258 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=275 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/C/ca25372fb32b48c08a5c970a1d76c719 2023-03-29 03:56:25,263 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/A/4bf75005213b4e6b88cafa073e2e8d13 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/4bf75005213b4e6b88cafa073e2e8d13 2023-03-29 03:56:25,266 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/4bf75005213b4e6b88cafa073e2e8d13, entries=150, sequenceid=275, filesize=25.4 K 2023-03-29 03:56:25,267 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/B/32523715f3fb4f3694681eb8024aa11d as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/32523715f3fb4f3694681eb8024aa11d 2023-03-29 03:56:25,271 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/32523715f3fb4f3694681eb8024aa11d, entries=150, sequenceid=275, filesize=11.9 K 2023-03-29 03:56:25,271 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/C/ca25372fb32b48c08a5c970a1d76c719 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/ca25372fb32b48c08a5c970a1d76c719 2023-03-29 03:56:25,281 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/ca25372fb32b48c08a5c970a1d76c719, entries=150, sequenceid=275, filesize=11.9 K 2023-03-29 03:56:25,282 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 94b16ca160750195aa137348cdb80ba9 in 872ms, sequenceid=275, compaction requested=false 2023-03-29 03:56:25,282 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 94b16ca160750195aa137348cdb80ba9: 2023-03-29 03:56:25,282 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:25,282 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:56:25,282 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:56:25,282 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:25,282 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:56:25,282 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:56:25,284 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:25,284 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:25,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:25,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:25,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:25,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:25,284 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:25,284 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:56:25,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:25,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:25,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:25,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:25,286 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:56:25,286 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@645612c9[Count = 0] remaining members to acquire global barrier 2023-03-29 03:56:25,286 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:56:25,286 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:25,287 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:25,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:25,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:25,287 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:56:25,287 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:56:25,287 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:56:25,287 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:25,287 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:56:25,289 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:25,289 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:56:25,289 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:25,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:25,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:25,289 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:56:25,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:25,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:25,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:25,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:25,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:25,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:25,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:25,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:25,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:56:25,291 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:56:25,291 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:56:25,291 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:56:25,291 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:56:25,291 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:56:25,292 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:56:25,296 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:25,296 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:25,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:25,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:25,296 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:25,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:25,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:25,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:25,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:25,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:25,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:25,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:25,297 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:25,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:25,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:25,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:25,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:25,299 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:25,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:25,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:25,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:25,302 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:25,302 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:25,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:25,302 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:56:25,302 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:25,302 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:56:25,302 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:56:25,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:25,302 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:25,302 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:25,302 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:25,302 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:25,302 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:25,302 DEBUG [Thread-4303] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:56:25,302 DEBUG [Thread-4303] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:56:25,302 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:25,303 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:25,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:25,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:25,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on 94b16ca160750195aa137348cdb80ba9 2023-03-29 03:56:25,332 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 94b16ca160750195aa137348cdb80ba9 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-03-29 03:56:25,333 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94b16ca160750195aa137348cdb80ba9, store=A 2023-03-29 03:56:25,333 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:56:25,333 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94b16ca160750195aa137348cdb80ba9, store=B 2023-03-29 03:56:25,333 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:56:25,333 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94b16ca160750195aa137348cdb80ba9, store=C 2023-03-29 03:56:25,333 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:56:25,351 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230329f7525d37bc264c62aebd7132e24f9e17 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230329f7525d37bc264c62aebd7132e24f9e17 2023-03-29 03:56:25,352 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:25,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37674 deadline: 1680062245347, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:25,352 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:25,352 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:25,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37666 deadline: 1680062245348, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:25,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37712 deadline: 1680062245348, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:25,352 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:25,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37670 deadline: 1680062245349, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:25,355 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:25,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37652 deadline: 1680062245352, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:25,365 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=294, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/A/fd56c068636b41b88a44c4782d008ef1 2023-03-29 03:56:25,381 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=294 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/B/31c17bc54c8c4265bfae30cf3621e3fe 2023-03-29 03:56:25,391 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=294 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/C/de05caeb70ba4557bf484790fe90404b 2023-03-29 03:56:25,395 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/A/fd56c068636b41b88a44c4782d008ef1 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/fd56c068636b41b88a44c4782d008ef1 2023-03-29 03:56:25,398 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/fd56c068636b41b88a44c4782d008ef1, entries=200, sequenceid=294, filesize=32.3 K 2023-03-29 03:56:25,399 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/B/31c17bc54c8c4265bfae30cf3621e3fe as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/31c17bc54c8c4265bfae30cf3621e3fe 2023-03-29 03:56:25,403 DEBUG [Thread-4303] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:56:25,403 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/31c17bc54c8c4265bfae30cf3621e3fe, entries=150, sequenceid=294, filesize=11.9 K 2023-03-29 03:56:25,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:56:25,404 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/C/de05caeb70ba4557bf484790fe90404b as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/de05caeb70ba4557bf484790fe90404b 2023-03-29 03:56:25,409 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:56:25,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:56:25,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:56:25,413 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:56:25,413 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:25,413 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:56:25,413 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:56:25,413 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:25,413 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:25,414 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/de05caeb70ba4557bf484790fe90404b, entries=150, sequenceid=294, filesize=11.9 K 2023-03-29 03:56:25,414 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=100.63 KB/103050 for 94b16ca160750195aa137348cdb80ba9 in 82ms, sequenceid=294, compaction requested=true 2023-03-29 03:56:25,415 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 94b16ca160750195aa137348cdb80ba9: 2023-03-29 03:56:25,415 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:25,415 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:25,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:25,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:25,415 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:25,415 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:56:25,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:25,415 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:25,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:56:25,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:25,416 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:56:25,415 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:56:25,416 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:56:25,415 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-29 03:56:25,416 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-29 03:56:25,416 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:56:25,416 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:56:25,416 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:56:25,417 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:25,417 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:56:25,417 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:56:25,417 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:56:25,417 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:56:25,417 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:25,417 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. started... 2023-03-29 03:56:25,417 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 94b16ca160750195aa137348cdb80ba9 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-03-29 03:56:25,417 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 85454 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-29 03:56:25,417 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94b16ca160750195aa137348cdb80ba9, store=A 2023-03-29 03:56:25,417 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1912): 94b16ca160750195aa137348cdb80ba9/A is initiating minor compaction (all files) 2023-03-29 03:56:25,418 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:56:25,418 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 94b16ca160750195aa137348cdb80ba9/A in TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:25,418 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36834 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-29 03:56:25,418 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/f40864718506408c8d813380f86cead0, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/4bf75005213b4e6b88cafa073e2e8d13, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/fd56c068636b41b88a44c4782d008ef1] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp, totalSize=83.5 K 2023-03-29 03:56:25,418 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1912): 94b16ca160750195aa137348cdb80ba9/B is initiating minor compaction (all files) 2023-03-29 03:56:25,418 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 94b16ca160750195aa137348cdb80ba9/B in TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:25,418 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/37682f23d96a415593111dc008377dcd, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/32523715f3fb4f3694681eb8024aa11d, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/31c17bc54c8c4265bfae30cf3621e3fe] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp, totalSize=36.0 K 2023-03-29 03:56:25,418 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94b16ca160750195aa137348cdb80ba9, store=B 2023-03-29 03:56:25,418 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting f40864718506408c8d813380f86cead0, keycount=150, bloomtype=ROW, size=25.7 K, encoding=NONE, compression=NONE, seqNum=257, earliestPutTs=1680062182755 2023-03-29 03:56:25,418 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:56:25,418 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 37682f23d96a415593111dc008377dcd, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=257, earliestPutTs=1680062182755 2023-03-29 03:56:25,418 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94b16ca160750195aa137348cdb80ba9, store=C 2023-03-29 03:56:25,419 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:56:25,419 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 4bf75005213b4e6b88cafa073e2e8d13, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=275, earliestPutTs=1680062183076 2023-03-29 03:56:25,419 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 32523715f3fb4f3694681eb8024aa11d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=275, earliestPutTs=1680062183076 2023-03-29 03:56:25,419 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting fd56c068636b41b88a44c4782d008ef1, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=294, earliestPutTs=1680062185201 2023-03-29 03:56:25,419 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 31c17bc54c8c4265bfae30cf3621e3fe, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=294, earliestPutTs=1680062185229 2023-03-29 03:56:25,430 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] throttle.PressureAwareThroughputController(145): 94b16ca160750195aa137348cdb80ba9#B#compaction#631 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:56:25,453 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] throttle.PressureAwareThroughputController(145): 94b16ca160750195aa137348cdb80ba9#A#compaction#630 average throughput is 0.79 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:56:25,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on 94b16ca160750195aa137348cdb80ba9 2023-03-29 03:56:25,457 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. as already flushing 2023-03-29 03:56:25,469 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:25,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37670 deadline: 1680062245465, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:25,469 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:25,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37712 deadline: 1680062245465, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:25,469 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:25,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37652 deadline: 1680062245466, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:25,471 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:25,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37674 deadline: 1680062245467, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:25,471 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:25,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37666 deadline: 1680062245468, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:25,472 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230329fe00076e78f840bf8a0fc1e20b1d67cd to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230329fe00076e78f840bf8a0fc1e20b1d67cd 2023-03-29 03:56:25,483 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=312, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/A/c730b09e1c5e4d9a8983539574402633 2023-03-29 03:56:25,499 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/A/2bb3bb5a36a34ebda2648f98e6f9729e as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/2bb3bb5a36a34ebda2648f98e6f9729e 2023-03-29 03:56:25,503 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=312 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/B/14f7ea675c654119ad69b22f47ed2a2d 2023-03-29 03:56:25,504 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 94b16ca160750195aa137348cdb80ba9/A of 94b16ca160750195aa137348cdb80ba9 into 2bb3bb5a36a34ebda2648f98e6f9729e(size=25.9 K), total size for store is 25.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:56:25,504 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 94b16ca160750195aa137348cdb80ba9: 2023-03-29 03:56:25,504 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9., storeName=94b16ca160750195aa137348cdb80ba9/A, priority=13, startTime=1680062185415; duration=0sec 2023-03-29 03:56:25,504 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:56:25,504 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-29 03:56:25,505 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36834 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-29 03:56:25,506 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1912): 94b16ca160750195aa137348cdb80ba9/C is initiating minor compaction (all files) 2023-03-29 03:56:25,506 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 94b16ca160750195aa137348cdb80ba9/C in TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:25,506 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/d284df99f2824152baff1fae57af725a, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/ca25372fb32b48c08a5c970a1d76c719, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/de05caeb70ba4557bf484790fe90404b] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp, totalSize=36.0 K 2023-03-29 03:56:25,506 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting d284df99f2824152baff1fae57af725a, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=257, earliestPutTs=1680062182755 2023-03-29 03:56:25,508 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting ca25372fb32b48c08a5c970a1d76c719, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=275, earliestPutTs=1680062183076 2023-03-29 03:56:25,508 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting de05caeb70ba4557bf484790fe90404b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=294, earliestPutTs=1680062185229 2023-03-29 03:56:25,517 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] throttle.PressureAwareThroughputController(145): 94b16ca160750195aa137348cdb80ba9#C#compaction#635 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:56:25,520 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=312 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/C/40df171f6f5a4e4daa24a80563045fb3 2023-03-29 03:56:25,524 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/A/c730b09e1c5e4d9a8983539574402633 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/c730b09e1c5e4d9a8983539574402633 2023-03-29 03:56:25,530 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/c730b09e1c5e4d9a8983539574402633, entries=150, sequenceid=312, filesize=25.4 K 2023-03-29 03:56:25,530 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/C/bba0413ad9e041cf9e98da70608d23c3 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/bba0413ad9e041cf9e98da70608d23c3 2023-03-29 03:56:25,531 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/B/14f7ea675c654119ad69b22f47ed2a2d as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/14f7ea675c654119ad69b22f47ed2a2d 2023-03-29 03:56:25,535 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/14f7ea675c654119ad69b22f47ed2a2d, entries=150, sequenceid=312, filesize=11.9 K 2023-03-29 03:56:25,535 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 94b16ca160750195aa137348cdb80ba9/C of 94b16ca160750195aa137348cdb80ba9 into bba0413ad9e041cf9e98da70608d23c3(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:56:25,535 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 94b16ca160750195aa137348cdb80ba9: 2023-03-29 03:56:25,536 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9., storeName=94b16ca160750195aa137348cdb80ba9/C, priority=13, startTime=1680062185416; duration=0sec 2023-03-29 03:56:25,536 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/C/40df171f6f5a4e4daa24a80563045fb3 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/40df171f6f5a4e4daa24a80563045fb3 2023-03-29 03:56:25,536 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:56:25,539 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/40df171f6f5a4e4daa24a80563045fb3, entries=150, sequenceid=312, filesize=11.9 K 2023-03-29 03:56:25,539 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 94b16ca160750195aa137348cdb80ba9 in 122ms, sequenceid=312, compaction requested=false 2023-03-29 03:56:25,539 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 94b16ca160750195aa137348cdb80ba9: 2023-03-29 03:56:25,540 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:25,540 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:56:25,540 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:56:25,540 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:25,540 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:56:25,540 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:56:25,541 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:25,541 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:25,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:25,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:25,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:25,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:25,542 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:25,542 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:56:25,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:25,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:25,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:25,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:25,543 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:56:25,543 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4fa8fb32[Count = 0] remaining members to acquire global barrier 2023-03-29 03:56:25,543 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:56:25,543 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:25,545 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:25,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:25,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:25,545 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:56:25,545 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:56:25,545 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:56:25,545 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:25,545 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:56:25,547 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:56:25,547 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:25,547 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:25,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:25,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:25,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:25,547 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:56:25,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:25,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:25,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:25,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:25,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:25,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:25,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:25,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:56:25,550 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:56:25,550 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:56:25,550 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:56:25,550 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:56:25,550 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:56:25,550 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:56:25,557 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:25,557 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:25,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:25,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:25,557 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:25,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:25,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:25,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:25,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:25,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:25,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:25,558 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:25,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:25,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:25,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:25,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:25,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:25,560 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:25,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:25,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:25,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:25,562 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:25,563 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:25,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:25,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:25,563 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:25,563 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:25,563 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:25,563 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:25,563 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:56:25,563 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:56:25,563 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:25,563 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:25,563 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:25,563 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:25,563 DEBUG [Thread-4303] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:56:25,563 DEBUG [Thread-4303] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:56:25,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:25,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:25,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on 94b16ca160750195aa137348cdb80ba9 2023-03-29 03:56:25,572 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 94b16ca160750195aa137348cdb80ba9 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-03-29 03:56:25,573 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94b16ca160750195aa137348cdb80ba9, store=A 2023-03-29 03:56:25,573 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:56:25,573 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94b16ca160750195aa137348cdb80ba9, store=B 2023-03-29 03:56:25,573 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:56:25,573 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94b16ca160750195aa137348cdb80ba9, store=C 2023-03-29 03:56:25,573 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:56:25,586 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:25,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37712 deadline: 1680062245582, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:25,586 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:25,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37652 deadline: 1680062245583, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:25,587 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:25,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37670 deadline: 1680062245583, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:25,588 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:25,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37674 deadline: 1680062245584, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:25,588 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:25,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37666 deadline: 1680062245585, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:25,590 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230329a4fe157ea7984e938c3f47c9ef34c8da to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230329a4fe157ea7984e938c3f47c9ef34c8da 2023-03-29 03:56:25,618 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=329, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/A/a4b76454c8f14de0bcbac98e0382d22b 2023-03-29 03:56:25,633 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=329 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/B/782ec38b1b984fbeb2d1995ba4db5b8d 2023-03-29 03:56:25,645 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=329 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/C/5928c0f34fb54229b579133bed9e3881 2023-03-29 03:56:25,649 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/A/a4b76454c8f14de0bcbac98e0382d22b as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/a4b76454c8f14de0bcbac98e0382d22b 2023-03-29 03:56:25,652 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/a4b76454c8f14de0bcbac98e0382d22b, entries=150, sequenceid=329, filesize=25.4 K 2023-03-29 03:56:25,653 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/B/782ec38b1b984fbeb2d1995ba4db5b8d as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/782ec38b1b984fbeb2d1995ba4db5b8d 2023-03-29 03:56:25,656 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/782ec38b1b984fbeb2d1995ba4db5b8d, entries=150, sequenceid=329, filesize=11.9 K 2023-03-29 03:56:25,657 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/C/5928c0f34fb54229b579133bed9e3881 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/5928c0f34fb54229b579133bed9e3881 2023-03-29 03:56:25,660 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/5928c0f34fb54229b579133bed9e3881, entries=150, sequenceid=329, filesize=11.9 K 2023-03-29 03:56:25,660 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 94b16ca160750195aa137348cdb80ba9 in 88ms, sequenceid=329, compaction requested=true 2023-03-29 03:56:25,660 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 94b16ca160750195aa137348cdb80ba9: 2023-03-29 03:56:25,660 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:56:25,660 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:56:25,660 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-29 03:56:25,660 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-29 03:56:25,661 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 78616 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-29 03:56:25,661 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1912): 94b16ca160750195aa137348cdb80ba9/A is initiating minor compaction (all files) 2023-03-29 03:56:25,661 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 94b16ca160750195aa137348cdb80ba9/A in TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:25,661 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/2bb3bb5a36a34ebda2648f98e6f9729e, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/c730b09e1c5e4d9a8983539574402633, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/a4b76454c8f14de0bcbac98e0382d22b] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp, totalSize=76.8 K 2023-03-29 03:56:25,662 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 2bb3bb5a36a34ebda2648f98e6f9729e, keycount=150, bloomtype=ROW, size=25.9 K, encoding=NONE, compression=NONE, seqNum=294, earliestPutTs=1680062185229 2023-03-29 03:56:25,662 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting c730b09e1c5e4d9a8983539574402633, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=312, earliestPutTs=1680062185345 2023-03-29 03:56:25,663 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting a4b76454c8f14de0bcbac98e0382d22b, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=329, earliestPutTs=1680062185463 2023-03-29 03:56:25,663 DEBUG [Thread-4303] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:56:25,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:56:25,666 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:56:25,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:56:25,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:56:25,667 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:56:25,667 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:25,667 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:56:25,667 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:56:25,667 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:25,668 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:25,670 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:25,670 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:25,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:25,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:25,670 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:25,670 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:56:25,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:25,670 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:25,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:56:25,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:25,671 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:56:25,672 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:56:25,672 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:56:25,672 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:56:25,672 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:25,672 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:56:25,676 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:56:25,676 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:56:25,676 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:25,676 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. started... 2023-03-29 03:56:25,676 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 94b16ca160750195aa137348cdb80ba9 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-03-29 03:56:25,676 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94b16ca160750195aa137348cdb80ba9, store=A 2023-03-29 03:56:25,676 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:56:25,676 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94b16ca160750195aa137348cdb80ba9, store=B 2023-03-29 03:56:25,676 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:56:25,676 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94b16ca160750195aa137348cdb80ba9, store=C 2023-03-29 03:56:25,676 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:56:25,681 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] throttle.PressureAwareThroughputController(145): 94b16ca160750195aa137348cdb80ba9#A#compaction#639 average throughput is 1.52 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:56:25,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on 94b16ca160750195aa137348cdb80ba9 2023-03-29 03:56:25,690 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. as already flushing 2023-03-29 03:56:25,702 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032966df0757043f41ea8d36b2459ec11471 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032966df0757043f41ea8d36b2459ec11471 2023-03-29 03:56:25,709 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=346, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/A/aee92c8248ce40e3a06f1cd5c14af621 2023-03-29 03:56:25,716 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=346 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/B/abfa254bacad4eeaac55353cf184f95a 2023-03-29 03:56:25,728 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:25,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37670 deadline: 1680062245724, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:25,728 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:25,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37674 deadline: 1680062245724, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:25,728 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=346 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/C/22ffbecf3e004da5aada22a0162d1473 2023-03-29 03:56:25,728 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:25,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37666 deadline: 1680062245724, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:25,729 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:25,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37652 deadline: 1680062245725, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:25,732 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:25,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37712 deadline: 1680062245728, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:25,734 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/A/aee92c8248ce40e3a06f1cd5c14af621 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/aee92c8248ce40e3a06f1cd5c14af621 2023-03-29 03:56:25,738 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/aee92c8248ce40e3a06f1cd5c14af621, entries=150, sequenceid=346, filesize=25.4 K 2023-03-29 03:56:25,738 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/B/abfa254bacad4eeaac55353cf184f95a as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/abfa254bacad4eeaac55353cf184f95a 2023-03-29 03:56:25,743 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/abfa254bacad4eeaac55353cf184f95a, entries=150, sequenceid=346, filesize=11.9 K 2023-03-29 03:56:25,743 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/C/22ffbecf3e004da5aada22a0162d1473 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/22ffbecf3e004da5aada22a0162d1473 2023-03-29 03:56:25,747 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/22ffbecf3e004da5aada22a0162d1473, entries=150, sequenceid=346, filesize=11.9 K 2023-03-29 03:56:25,747 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 94b16ca160750195aa137348cdb80ba9 in 71ms, sequenceid=346, compaction requested=true 2023-03-29 03:56:25,748 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 94b16ca160750195aa137348cdb80ba9: 2023-03-29 03:56:25,748 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:25,748 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:56:25,748 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:56:25,748 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:25,748 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:56:25,748 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:56:25,750 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:25,756 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:25,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:25,756 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:25,756 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:56:25,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:25,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:25,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:25,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:25,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:25,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:25,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:25,758 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:56:25,758 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@733a2b00[Count = 0] remaining members to acquire global barrier 2023-03-29 03:56:25,758 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:56:25,758 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:25,760 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:25,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:25,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:25,760 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:56:25,760 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:56:25,760 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:56:25,760 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:25,760 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:56:25,764 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:56:25,765 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:25,764 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:25,765 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:56:25,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:25,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:25,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:25,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:25,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:25,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:25,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:25,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:25,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:25,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:25,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:56:25,768 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:56:25,768 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:56:25,768 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:56:25,768 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:56:25,768 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:56:25,768 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:56:25,770 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:25,770 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:25,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:25,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:25,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:25,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:25,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:25,770 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:25,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:25,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:25,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:25,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:25,771 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:25,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:25,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:25,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:25,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:25,772 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:25,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:25,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:25,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:25,775 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:25,775 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:25,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:25,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:25,775 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:25,775 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:25,775 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:56:25,776 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:56:25,776 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:25,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:25,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:25,776 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:25,776 DEBUG [Thread-4303] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:56:25,776 DEBUG [Thread-4303] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:56:25,776 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:25,776 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:25,776 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:25,776 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:25,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on 94b16ca160750195aa137348cdb80ba9 2023-03-29 03:56:25,832 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 94b16ca160750195aa137348cdb80ba9 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-03-29 03:56:25,833 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94b16ca160750195aa137348cdb80ba9, store=A 2023-03-29 03:56:25,833 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:56:25,833 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94b16ca160750195aa137348cdb80ba9, store=B 2023-03-29 03:56:25,833 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:56:25,833 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94b16ca160750195aa137348cdb80ba9, store=C 2023-03-29 03:56:25,833 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:56:25,867 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/B/877e2162b8a4487e8ef12c4da960d6fd as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/877e2162b8a4487e8ef12c4da960d6fd 2023-03-29 03:56:25,870 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:25,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37674 deadline: 1680062245865, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:25,871 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 94b16ca160750195aa137348cdb80ba9/B of 94b16ca160750195aa137348cdb80ba9 into 877e2162b8a4487e8ef12c4da960d6fd(size=12.4 K), total size for store is 48.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:56:25,871 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 94b16ca160750195aa137348cdb80ba9: 2023-03-29 03:56:25,871 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9., storeName=94b16ca160750195aa137348cdb80ba9/B, priority=13, startTime=1680062185416; duration=0sec 2023-03-29 03:56:25,871 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:25,871 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-29 03:56:25,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37670 deadline: 1680062245866, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:25,872 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-29 03:56:25,872 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:25,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37652 deadline: 1680062245867, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:25,872 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:25,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37712 deadline: 1680062245867, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:25,872 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:25,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37666 deadline: 1680062245867, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:25,873 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49222 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-29 03:56:25,873 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1912): 94b16ca160750195aa137348cdb80ba9/B is initiating minor compaction (all files) 2023-03-29 03:56:25,873 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 94b16ca160750195aa137348cdb80ba9/B in TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:25,873 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/877e2162b8a4487e8ef12c4da960d6fd, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/14f7ea675c654119ad69b22f47ed2a2d, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/782ec38b1b984fbeb2d1995ba4db5b8d, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/abfa254bacad4eeaac55353cf184f95a] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp, totalSize=48.1 K 2023-03-29 03:56:25,874 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 877e2162b8a4487e8ef12c4da960d6fd, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=294, earliestPutTs=1680062185229 2023-03-29 03:56:25,874 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 14f7ea675c654119ad69b22f47ed2a2d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=312, earliestPutTs=1680062185345 2023-03-29 03:56:25,874 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 782ec38b1b984fbeb2d1995ba4db5b8d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=329, earliestPutTs=1680062185463 2023-03-29 03:56:25,874 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting abfa254bacad4eeaac55353cf184f95a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=346, earliestPutTs=1680062185579 2023-03-29 03:56:25,876 DEBUG [Thread-4303] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:56:25,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:56:25,878 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:56:25,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:56:25,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:56:25,880 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:56:25,880 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:25,880 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:56:25,880 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:56:25,880 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:25,880 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:25,882 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] throttle.PressureAwareThroughputController(145): 94b16ca160750195aa137348cdb80ba9#B#compaction#644 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:56:25,882 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:25,882 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:25,882 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:56:25,882 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:25,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:25,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:25,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:25,883 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:25,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:56:25,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:25,884 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:56:25,888 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:56:25,888 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:56:25,888 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:56:25,888 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:25,888 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:56:25,893 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:56:25,893 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:56:25,893 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:25,893 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. started... 2023-03-29 03:56:25,893 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. as already flushing 2023-03-29 03:56:25,893 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:25,893 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:56:25,893 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:56:25,893 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:25,896 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:56:25,896 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:56:25,897 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:25,897 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:25,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:25,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:25,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:25,898 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:25,898 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:56:25,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:25,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:25,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:25,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:25,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:25,900 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:56:25,901 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@61de0784[Count = 0] remaining members to acquire global barrier 2023-03-29 03:56:25,901 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:56:25,901 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:25,902 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:25,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:25,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:25,902 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:56:25,902 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:56:25,902 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:56:25,903 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:25,903 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:56:25,903 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/B/08961dda88994f33b09ce83318cf63c0 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/08961dda88994f33b09ce83318cf63c0 2023-03-29 03:56:25,905 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:25,905 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:56:25,905 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:25,905 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:56:25,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:25,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:25,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:25,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:25,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:25,907 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 94b16ca160750195aa137348cdb80ba9/B of 94b16ca160750195aa137348cdb80ba9 into 08961dda88994f33b09ce83318cf63c0(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:56:25,907 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 94b16ca160750195aa137348cdb80ba9: 2023-03-29 03:56:25,907 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9., storeName=94b16ca160750195aa137348cdb80ba9/B, priority=12, startTime=1680062185660; duration=0sec 2023-03-29 03:56:25,907 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:56:25,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:25,908 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-29 03:56:25,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:25,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:25,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:25,909 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49222 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-29 03:56:25,909 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1912): 94b16ca160750195aa137348cdb80ba9/C is initiating minor compaction (all files) 2023-03-29 03:56:25,909 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 94b16ca160750195aa137348cdb80ba9/C in TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:25,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:25,909 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/bba0413ad9e041cf9e98da70608d23c3, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/40df171f6f5a4e4daa24a80563045fb3, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/5928c0f34fb54229b579133bed9e3881, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/22ffbecf3e004da5aada22a0162d1473] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp, totalSize=48.1 K 2023-03-29 03:56:25,909 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting bba0413ad9e041cf9e98da70608d23c3, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=294, earliestPutTs=1680062185229 2023-03-29 03:56:25,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:56:25,910 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:56:25,910 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 40df171f6f5a4e4daa24a80563045fb3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=312, earliestPutTs=1680062185345 2023-03-29 03:56:25,910 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:56:25,910 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:56:25,910 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:56:25,910 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:56:25,910 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:56:25,910 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 5928c0f34fb54229b579133bed9e3881, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=329, earliestPutTs=1680062185463 2023-03-29 03:56:25,911 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 22ffbecf3e004da5aada22a0162d1473, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=346, earliestPutTs=1680062185579 2023-03-29 03:56:25,912 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:25,912 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:25,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:25,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:25,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:25,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:25,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:25,912 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:25,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:25,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:25,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:25,913 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:25,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:25,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:25,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:25,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:25,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:25,916 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:25,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:25,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:25,918 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] throttle.PressureAwareThroughputController(145): 94b16ca160750195aa137348cdb80ba9#C#compaction#645 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:56:25,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:25,922 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:25,922 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:25,922 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:25,922 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:25,922 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:25,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:25,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:25,922 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:25,922 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:56:25,923 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:56:25,922 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:25,923 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:25,923 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:25,923 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:25,923 DEBUG [Thread-4303] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:56:25,923 DEBUG [Thread-4303] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:56:25,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:25,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:25,935 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/C/9997ca2063044d3b89fedb3f63e599c4 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/9997ca2063044d3b89fedb3f63e599c4 2023-03-29 03:56:25,941 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 94b16ca160750195aa137348cdb80ba9/C of 94b16ca160750195aa137348cdb80ba9 into 9997ca2063044d3b89fedb3f63e599c4(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:56:25,942 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 94b16ca160750195aa137348cdb80ba9: 2023-03-29 03:56:25,942 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9., storeName=94b16ca160750195aa137348cdb80ba9/C, priority=12, startTime=1680062185660; duration=0sec 2023-03-29 03:56:25,942 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:56:25,972 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:25,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37674 deadline: 1680062245971, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:25,974 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:25,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37652 deadline: 1680062245973, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:25,974 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:25,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37670 deadline: 1680062245973, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:25,975 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:25,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37712 deadline: 1680062245973, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:25,975 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:25,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37666 deadline: 1680062245973, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,023 DEBUG [Thread-4303] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:56:26,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:56:26,025 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:56:26,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:56:26,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:56:26,027 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:56:26,027 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:26,027 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:56:26,027 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:56:26,028 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:26,028 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:26,030 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,030 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:26,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:26,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:26,030 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,030 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:56:26,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:26,030 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:26,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:56:26,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:26,031 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:56:26,031 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:56:26,031 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:56:26,031 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:56:26,031 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:26,031 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:56:26,031 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:56:26,031 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:56:26,031 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:26,031 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. started... 2023-03-29 03:56:26,032 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. as already flushing 2023-03-29 03:56:26,032 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:26,032 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:56:26,032 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:56:26,032 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,032 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:56:26,032 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:56:26,038 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,038 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:26,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:26,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:26,039 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:26,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:26,039 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:56:26,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:26,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:26,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:26,040 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:56:26,040 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@35ca812c[Count = 0] remaining members to acquire global barrier 2023-03-29 03:56:26,040 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:56:26,040 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:26,042 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:26,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:26,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:26,042 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:56:26,042 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:56:26,042 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,042 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:56:26,042 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:56:26,044 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,044 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:56:26,044 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:26,044 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:56:26,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:26,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:26,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:26,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:26,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:26,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:26,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:26,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:56:26,047 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:56:26,047 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:56:26,047 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:56:26,047 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:56:26,047 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:56:26,047 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:56:26,049 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:26,049 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:26,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:26,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:26,049 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,049 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:26,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:26,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:26,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:26,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:26,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:26,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:26,050 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:26,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:26,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:26,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:26,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,058 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,058 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:26,058 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:26,058 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,058 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:26,058 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:26,058 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:56:26,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:26,058 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:26,058 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:56:26,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:26,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:26,058 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:26,058 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:26,058 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:26,058 DEBUG [Thread-4303] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:56:26,058 DEBUG [Thread-4303] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:56:26,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:26,059 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:56:26,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:26,117 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/A/11ce9d02c9c941a4a9277d2ef2992926 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/11ce9d02c9c941a4a9277d2ef2992926 2023-03-29 03:56:26,121 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 94b16ca160750195aa137348cdb80ba9/A of 94b16ca160750195aa137348cdb80ba9 into 11ce9d02c9c941a4a9277d2ef2992926(size=26.0 K), total size for store is 51.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:56:26,122 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 94b16ca160750195aa137348cdb80ba9: 2023-03-29 03:56:26,122 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9., storeName=94b16ca160750195aa137348cdb80ba9/A, priority=13, startTime=1680062185660; duration=0sec 2023-03-29 03:56:26,122 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:56:26,159 DEBUG [Thread-4303] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:56:26,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:56:26,161 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:56:26,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:56:26,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:56:26,162 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:56:26,162 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:26,162 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:56:26,162 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:56:26,162 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:26,163 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:26,164 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:26,164 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:26,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:26,164 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,164 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:56:26,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:26,165 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:26,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:56:26,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:26,165 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:56:26,166 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:56:26,166 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:56:26,166 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:56:26,166 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:26,166 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:56:26,166 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:56:26,166 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:56:26,166 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:26,166 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. started... 2023-03-29 03:56:26,166 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. as already flushing 2023-03-29 03:56:26,166 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:26,166 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:56:26,166 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:56:26,166 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,166 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:56:26,167 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:56:26,168 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:26,168 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:26,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:26,168 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:26,168 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:56:26,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:26,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:26,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:26,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:26,170 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:56:26,170 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d339dcc[Count = 0] remaining members to acquire global barrier 2023-03-29 03:56:26,170 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:56:26,170 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:26,171 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:26,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:26,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:26,171 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:56:26,171 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:56:26,171 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:56:26,171 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,171 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:56:26,175 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:26,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37674 deadline: 1680062246173, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,175 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,175 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:56:26,175 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:26,175 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:56:26,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:26,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:26,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:26,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:26,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:26,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:26,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:26,178 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:26,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37712 deadline: 1680062246176, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,178 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:26,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37666 deadline: 1680062246176, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,178 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:26,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37652 deadline: 1680062246176, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,178 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:26,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37670 deadline: 1680062246176, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:56:26,179 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:56:26,179 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:56:26,179 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:56:26,179 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:56:26,179 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:56:26,179 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:56:26,180 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:26,180 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:26,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:26,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:26,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:26,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:26,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:26,180 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:26,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:26,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:26,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:26,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:26,181 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:26,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:26,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:26,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,182 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:26,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:26,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,188 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:26,189 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,189 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:26,189 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:26,189 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:56:26,189 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:56:26,189 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:26,189 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:26,189 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,189 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:26,189 DEBUG [Thread-4303] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:56:26,189 DEBUG [Thread-4303] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:56:26,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:26,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:26,189 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:26,189 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:26,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:26,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:26,249 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032986cc729bf52f46f8b8c3d74f1b4a7e1c to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032986cc729bf52f46f8b8c3d74f1b4a7e1c 2023-03-29 03:56:26,289 DEBUG [Thread-4303] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:56:26,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:56:26,291 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:56:26,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:56:26,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:56:26,292 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:56:26,292 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:26,292 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:56:26,292 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:56:26,293 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:26,293 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:26,294 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:26,294 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:26,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:26,294 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,294 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:56:26,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:26,295 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:26,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:56:26,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:26,295 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:56:26,296 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:56:26,296 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:56:26,296 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:56:26,296 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:26,296 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:56:26,296 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:56:26,296 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:56:26,296 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:26,296 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. started... 2023-03-29 03:56:26,296 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. as already flushing 2023-03-29 03:56:26,296 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:26,296 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:56:26,296 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:56:26,296 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,296 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:56:26,296 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:56:26,299 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,299 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:26,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:26,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:26,299 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:26,299 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:56:26,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:26,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:26,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:26,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:26,302 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:56:26,302 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1ff3c341[Count = 0] remaining members to acquire global barrier 2023-03-29 03:56:26,302 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:56:26,302 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:26,303 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:26,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:26,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:26,303 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:56:26,303 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:56:26,303 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:56:26,303 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,304 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:56:26,304 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,304 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:56:26,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:26,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:26,304 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:26,305 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:56:26,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:26,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:26,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:26,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:26,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:26,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:56:26,307 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:56:26,307 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:56:26,307 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:56:26,307 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:56:26,307 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:56:26,307 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:56:26,309 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:26,309 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:26,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:26,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:26,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:26,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:26,309 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:26,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:26,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:26,310 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:26,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:26,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:26,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:26,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:26,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:26,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,311 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:26,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:26,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,315 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,315 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:26,315 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:26,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:26,315 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:26,315 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:26,315 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:56:26,316 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:56:26,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:26,315 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:26,316 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,316 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:26,316 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:26,316 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:26,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:26,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:26,316 DEBUG [Thread-4303] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:56:26,316 DEBUG [Thread-4303] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:56:26,417 DEBUG [Thread-4303] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:56:26,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:56:26,419 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:56:26,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:56:26,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:56:26,420 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:56:26,420 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:26,420 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:56:26,420 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:56:26,420 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:26,420 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:26,421 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:26,421 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:26,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:26,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:26,422 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,422 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:56:26,422 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:26,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:56:26,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:26,423 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:56:26,423 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:56:26,423 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:56:26,423 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:56:26,423 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:26,423 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:56:26,423 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:56:26,423 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:56:26,423 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:26,423 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. started... 2023-03-29 03:56:26,424 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. as already flushing 2023-03-29 03:56:26,424 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:26,424 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:56:26,424 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:56:26,424 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,424 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:56:26,424 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:56:26,425 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:26,425 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:26,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:26,425 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:26,426 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:56:26,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:26,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:26,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:26,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:26,427 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:56:26,427 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e00051c[Count = 0] remaining members to acquire global barrier 2023-03-29 03:56:26,427 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:56:26,427 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:26,428 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:26,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:26,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:26,428 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:56:26,428 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:56:26,428 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:56:26,429 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,429 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:56:26,431 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:56:26,431 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:26,431 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,431 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:56:26,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:26,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:26,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:26,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:26,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:26,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:26,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:26,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:56:26,434 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:56:26,434 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:56:26,434 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:56:26,434 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:56:26,434 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:56:26,434 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:56:26,435 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:26,435 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:26,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:26,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:26,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:26,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:26,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:26,435 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:26,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:26,436 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:26,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:26,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:26,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:26,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:26,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:26,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,437 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:26,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:26,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,439 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,439 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:26,439 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:26,439 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:26,439 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:26,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:26,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:26,439 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:26,439 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:56:26,440 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:56:26,439 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,440 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:26,440 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:26,440 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:26,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:26,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:26,440 DEBUG [Thread-4303] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:56:26,440 DEBUG [Thread-4303] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:56:26,479 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:26,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37674 deadline: 1680062246477, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,480 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:26,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37712 deadline: 1680062246479, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,480 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:26,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37666 deadline: 1680062246479, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,482 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:26,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37652 deadline: 1680062246479, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,483 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:26,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37670 deadline: 1680062246481, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,540 DEBUG [Thread-4303] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:56:26,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:56:26,542 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:56:26,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:56:26,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:56:26,543 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:56:26,543 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:26,544 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:56:26,544 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:56:26,544 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:26,544 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:26,545 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:26,545 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:26,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:26,545 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,546 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:56:26,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:26,546 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:26,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:56:26,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:26,546 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:56:26,546 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:56:26,546 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:56:26,546 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:56:26,547 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:26,547 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:56:26,547 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:56:26,547 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:56:26,547 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:26,547 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. started... 2023-03-29 03:56:26,547 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. as already flushing 2023-03-29 03:56:26,547 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:26,547 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:56:26,547 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:56:26,547 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,547 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:56:26,547 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:56:26,550 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,550 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:26,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:26,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:26,550 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:26,550 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:56:26,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:26,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:26,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:26,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:26,552 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:56:26,552 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6fa22611[Count = 0] remaining members to acquire global barrier 2023-03-29 03:56:26,552 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:56:26,552 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:26,553 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:26,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:26,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:26,553 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:56:26,553 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:56:26,553 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:56:26,554 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,554 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:56:26,555 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:56:26,555 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:26,555 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,555 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:56:26,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:26,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:26,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:26,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:26,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:26,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:26,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:26,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:56:26,560 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:56:26,560 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:56:26,560 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:56:26,560 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:56:26,560 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:56:26,560 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:56:26,562 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:26,562 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:26,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:26,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:26,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:26,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:26,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:26,562 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:26,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:26,563 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:26,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:26,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:26,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:26,564 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:26,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:26,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:26,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:26,574 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:26,574 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:26,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:26,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:26,574 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:26,574 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:56:26,574 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:56:26,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:26,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:26,574 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,574 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:26,574 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:26,574 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,574 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:26,574 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:26,574 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:26,574 DEBUG [Thread-4303] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:56:26,574 DEBUG [Thread-4303] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:56:26,574 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:56:26,661 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=364, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/A/e4ef63ec149f4296a2037aa8e4069b30 2023-03-29 03:56:26,674 DEBUG [Thread-4303] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:56:26,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:56:26,684 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:56:26,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:56:26,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:56:26,685 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:56:26,686 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:26,686 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:56:26,686 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:56:26,686 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:26,686 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:26,687 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:26,687 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:26,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:26,688 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,688 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:56:26,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:26,689 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:26,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:56:26,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:26,689 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:56:26,690 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:56:26,690 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:56:26,690 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:56:26,690 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:26,690 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:56:26,692 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:56:26,692 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:26,692 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:56:26,692 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. started... 2023-03-29 03:56:26,692 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. as already flushing 2023-03-29 03:56:26,692 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:26,692 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:56:26,692 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:56:26,692 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,692 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:56:26,692 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:56:26,694 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,694 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:26,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:26,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:26,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:26,694 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:26,694 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:56:26,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:26,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:26,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:26,695 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:56:26,695 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4e5e0bd4[Count = 0] remaining members to acquire global barrier 2023-03-29 03:56:26,695 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:56:26,695 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:26,696 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:26,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:26,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:26,697 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:56:26,697 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:56:26,697 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:56:26,697 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,697 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:56:26,698 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,699 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:56:26,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:26,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:26,699 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:26,699 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:56:26,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:26,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:26,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:26,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:26,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:26,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:56:26,701 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:56:26,701 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:56:26,701 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:56:26,701 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:56:26,701 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:56:26,701 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:56:26,702 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:26,702 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:26,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:26,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:26,702 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:26,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:26,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:26,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:26,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:26,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:26,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:26,703 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:26,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:26,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:26,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:26,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,704 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:26,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:26,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,706 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:26,706 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,706 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:26,706 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:26,706 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:56:26,706 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:56:26,706 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:26,706 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:26,706 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,706 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:26,706 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:26,706 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:26,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:26,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:26,706 DEBUG [Thread-4303] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:56:26,706 DEBUG [Thread-4303] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:56:26,706 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:56:26,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:26,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:26,806 DEBUG [Thread-4303] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:56:26,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:56:26,809 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:56:26,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:56:26,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:56:26,810 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:56:26,810 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:26,810 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:56:26,810 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:56:26,811 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:26,811 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:26,812 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,812 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:26,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:26,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:26,812 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,812 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:56:26,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:26,812 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:26,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:56:26,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:26,813 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:56:26,813 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:56:26,813 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:56:26,813 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:56:26,813 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:26,813 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:56:26,814 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:56:26,814 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:56:26,814 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:26,814 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. started... 2023-03-29 03:56:26,814 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. as already flushing 2023-03-29 03:56:26,814 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:26,814 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:56:26,814 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:56:26,814 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,814 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:56:26,814 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:56:26,817 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:26,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:26,817 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:26,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:26,818 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:26,818 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:56:26,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:26,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:26,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:26,819 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:56:26,819 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f362095[Count = 0] remaining members to acquire global barrier 2023-03-29 03:56:26,819 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:56:26,819 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:26,820 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:26,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:26,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:26,820 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:56:26,821 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:56:26,821 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:56:26,821 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,821 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:56:26,822 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:56:26,822 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:26,822 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,822 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:56:26,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:26,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:26,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:26,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:26,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:26,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:26,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:26,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:56:26,824 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:56:26,824 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:56:26,824 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:56:26,824 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:56:26,824 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:56:26,824 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:56:26,825 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:26,826 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:26,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:26,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:26,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:26,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:26,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:26,826 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:26,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:26,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:26,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:26,826 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:26,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:26,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:26,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:26,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,827 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:26,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:26,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,835 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,835 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:26,835 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:26,835 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,835 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:26,835 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:26,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:26,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:26,835 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:26,835 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:56:26,835 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:56:26,835 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:26,836 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:26,836 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:26,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:26,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:26,836 DEBUG [Thread-4303] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:56:26,836 DEBUG [Thread-4303] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:56:26,936 DEBUG [Thread-4303] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:56:26,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:56:26,938 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:56:26,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:56:26,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:56:26,939 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:56:26,939 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:26,939 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:56:26,939 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:56:26,940 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:26,940 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:26,941 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,941 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:26,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:26,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:26,941 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,941 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:56:26,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:26,942 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:26,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:56:26,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:26,942 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:56:26,942 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:56:26,942 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:56:26,942 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:56:26,942 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:26,942 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:56:26,943 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:56:26,943 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:56:26,943 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:26,943 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. started... 2023-03-29 03:56:26,943 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. as already flushing 2023-03-29 03:56:26,943 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:26,943 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:56:26,943 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:56:26,943 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,943 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:56:26,943 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:56:26,948 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,948 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:26,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:26,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:26,948 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:26,948 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:56:26,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:26,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:26,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:26,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:26,950 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:56:26,950 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4110f48a[Count = 0] remaining members to acquire global barrier 2023-03-29 03:56:26,950 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:56:26,950 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:26,951 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:26,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:26,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:26,951 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:56:26,951 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,951 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:56:26,951 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:56:26,951 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:56:26,953 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:26,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:26,954 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:56:26,954 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:26,954 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:56:26,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:26,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:26,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:26,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:26,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:26,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:56:26,956 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:56:26,956 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:56:26,956 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:56:26,956 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:56:26,956 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:56:26,956 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:56:26,957 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:26,957 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:26,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:26,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:26,957 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:26,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:26,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:26,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:26,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:26,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:26,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:26,958 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:26,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:26,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:26,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:26,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,959 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:26,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:26,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,962 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:26,962 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,962 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:26,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:26,962 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:56:26,962 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:56:26,962 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:26,962 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:26,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:26,962 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:26,962 DEBUG [Thread-4303] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:56:26,962 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,962 DEBUG [Thread-4303] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:56:26,962 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:26,962 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:26,962 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:26,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:26,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:26,983 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:26,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37652 deadline: 1680062246983, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,984 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:26,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37712 deadline: 1680062246983, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,986 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:26,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37666 deadline: 1680062246984, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,986 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:26,986 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:26,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37674 deadline: 1680062246984, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:26,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37670 deadline: 1680062246985, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:27,062 DEBUG [Thread-4303] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:56:27,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:56:27,064 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:56:27,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:56:27,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:56:27,066 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:56:27,066 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:27,066 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:56:27,066 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:56:27,066 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:27,066 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:27,070 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=364 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/B/6c3d3f961160426aa3de2574eb4d22d6 2023-03-29 03:56:27,070 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:27,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:27,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:27,070 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:27,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:27,071 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:27,071 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:27,071 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:56:27,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:56:27,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:27,072 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:56:27,073 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:56:27,073 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:56:27,073 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:56:27,073 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:27,073 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:56:27,073 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:56:27,073 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:56:27,073 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:27,073 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. started... 2023-03-29 03:56:27,073 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. as already flushing 2023-03-29 03:56:27,073 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:27,076 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:56:27,076 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:56:27,076 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:27,076 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:56:27,076 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:56:27,079 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:27,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:27,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:27,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:27,079 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:27,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:27,081 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:27,081 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:56:27,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:27,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:27,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:27,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:27,082 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:56:27,082 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ffcd017[Count = 0] remaining members to acquire global barrier 2023-03-29 03:56:27,082 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:56:27,082 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:27,084 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:27,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:27,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:27,084 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:56:27,084 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:56:27,084 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:56:27,084 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:27,084 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:56:27,085 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=364 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/C/50a0006b926e457fa286a13cb3d0ff24 2023-03-29 03:56:27,085 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:27,085 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:56:27,086 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:27,086 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:56:27,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:27,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:27,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:27,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:27,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:27,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:27,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:27,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:27,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:27,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:27,089 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/A/e4ef63ec149f4296a2037aa8e4069b30 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/e4ef63ec149f4296a2037aa8e4069b30 2023-03-29 03:56:27,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:56:27,089 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:56:27,090 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:56:27,090 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:56:27,090 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:56:27,090 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:56:27,090 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:56:27,093 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:27,093 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:27,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:27,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:27,093 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:27,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:27,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:27,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:27,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:27,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:27,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:27,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:27,095 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:27,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:27,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:27,096 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/e4ef63ec149f4296a2037aa8e4069b30, entries=200, sequenceid=364, filesize=32.3 K 2023-03-29 03:56:27,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:27,097 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/B/6c3d3f961160426aa3de2574eb4d22d6 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/6c3d3f961160426aa3de2574eb4d22d6 2023-03-29 03:56:27,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:27,097 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:27,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:27,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:27,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:27,099 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:27,099 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:27,099 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:27,099 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:56:27,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:27,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:27,099 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:27,099 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:56:27,100 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:56:27,099 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:27,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:27,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:27,100 DEBUG [Thread-4303] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:56:27,100 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:27,100 DEBUG [Thread-4303] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:56:27,100 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:27,100 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:27,100 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:27,100 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:27,102 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/6c3d3f961160426aa3de2574eb4d22d6, entries=150, sequenceid=364, filesize=11.9 K 2023-03-29 03:56:27,103 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/C/50a0006b926e457fa286a13cb3d0ff24 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/50a0006b926e457fa286a13cb3d0ff24 2023-03-29 03:56:27,113 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/50a0006b926e457fa286a13cb3d0ff24, entries=150, sequenceid=364, filesize=11.9 K 2023-03-29 03:56:27,114 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 94b16ca160750195aa137348cdb80ba9 in 1282ms, sequenceid=364, compaction requested=true 2023-03-29 03:56:27,114 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 94b16ca160750195aa137348cdb80ba9: 2023-03-29 03:56:27,114 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:56:27,114 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-29 03:56:27,114 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-03-29 03:56:27,114 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:56:27,114 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:56:27,115 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-03-29 03:56:27,115 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-03-29 03:56:27,115 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. because compaction request was cancelled 2023-03-29 03:56:27,115 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-03-29 03:56:27,115 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 85758 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-29 03:56:27,115 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1912): 94b16ca160750195aa137348cdb80ba9/A is initiating minor compaction (all files) 2023-03-29 03:56:27,115 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 94b16ca160750195aa137348cdb80ba9/A in TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:27,115 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/11ce9d02c9c941a4a9277d2ef2992926, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/aee92c8248ce40e3a06f1cd5c14af621, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/e4ef63ec149f4296a2037aa8e4069b30] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp, totalSize=83.7 K 2023-03-29 03:56:27,116 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 11ce9d02c9c941a4a9277d2ef2992926, keycount=150, bloomtype=ROW, size=26.0 K, encoding=NONE, compression=NONE, seqNum=329, earliestPutTs=1680062185463 2023-03-29 03:56:27,116 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-03-29 03:56:27,116 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-03-29 03:56:27,116 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. because compaction request was cancelled 2023-03-29 03:56:27,116 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting aee92c8248ce40e3a06f1cd5c14af621, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=346, earliestPutTs=1680062185579 2023-03-29 03:56:27,117 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting e4ef63ec149f4296a2037aa8e4069b30, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=364, earliestPutTs=1680062185694 2023-03-29 03:56:27,125 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] throttle.PressureAwareThroughputController(145): 94b16ca160750195aa137348cdb80ba9#A#compaction#648 average throughput is 4.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:56:27,145 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/A/51570dbb8b0d45fa87eb33fd8e50f2ac as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/51570dbb8b0d45fa87eb33fd8e50f2ac 2023-03-29 03:56:27,149 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 94b16ca160750195aa137348cdb80ba9/A of 94b16ca160750195aa137348cdb80ba9 into 51570dbb8b0d45fa87eb33fd8e50f2ac(size=26.1 K), total size for store is 26.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:56:27,149 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 94b16ca160750195aa137348cdb80ba9: 2023-03-29 03:56:27,149 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9., storeName=94b16ca160750195aa137348cdb80ba9/A, priority=13, startTime=1680062187114; duration=0sec 2023-03-29 03:56:27,149 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:56:27,200 DEBUG [Thread-4303] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:56:27,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:56:27,202 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:56:27,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:56:27,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:56:27,204 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:56:27,204 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:27,204 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:56:27,204 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:56:27,204 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:27,204 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:27,206 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:27,206 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:27,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:27,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:27,206 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:27,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:27,206 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:56:27,206 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:27,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:56:27,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:27,207 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:56:27,207 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:56:27,207 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:56:27,207 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:56:27,207 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:27,207 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:56:27,208 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:56:27,208 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:56:27,208 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:27,208 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. started... 2023-03-29 03:56:27,208 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 94b16ca160750195aa137348cdb80ba9 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-03-29 03:56:27,208 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94b16ca160750195aa137348cdb80ba9, store=A 2023-03-29 03:56:27,208 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:56:27,208 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94b16ca160750195aa137348cdb80ba9, store=B 2023-03-29 03:56:27,208 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:56:27,208 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94b16ca160750195aa137348cdb80ba9, store=C 2023-03-29 03:56:27,208 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:56:27,228 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303291f9121152c5546c08f558b74e018bf4d to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303291f9121152c5546c08f558b74e018bf4d 2023-03-29 03:56:27,634 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=383, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/A/ce2387e240424455b65afa1be80c1acf 2023-03-29 03:56:27,643 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=383 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/B/0993640b669a4f09ae3a0cabfdc08c06 2023-03-29 03:56:27,653 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=383 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/C/47647c64df1d43358d3ae25f251add7d 2023-03-29 03:56:27,656 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/A/ce2387e240424455b65afa1be80c1acf as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/ce2387e240424455b65afa1be80c1acf 2023-03-29 03:56:27,660 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/ce2387e240424455b65afa1be80c1acf, entries=100, sequenceid=383, filesize=18.6 K 2023-03-29 03:56:27,660 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/B/0993640b669a4f09ae3a0cabfdc08c06 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/0993640b669a4f09ae3a0cabfdc08c06 2023-03-29 03:56:27,664 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/0993640b669a4f09ae3a0cabfdc08c06, entries=100, sequenceid=383, filesize=9.5 K 2023-03-29 03:56:27,665 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/C/47647c64df1d43358d3ae25f251add7d as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/47647c64df1d43358d3ae25f251add7d 2023-03-29 03:56:27,668 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/47647c64df1d43358d3ae25f251add7d, entries=100, sequenceid=383, filesize=9.5 K 2023-03-29 03:56:27,669 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=0 B/0 for 94b16ca160750195aa137348cdb80ba9 in 461ms, sequenceid=383, compaction requested=true 2023-03-29 03:56:27,669 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 94b16ca160750195aa137348cdb80ba9: 2023-03-29 03:56:27,669 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:27,669 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:56:27,669 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:56:27,669 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:27,669 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:56:27,669 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:56:27,671 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:27,671 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:27,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:27,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:27,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:27,671 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:27,671 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:56:27,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:27,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:27,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:27,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:27,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:27,673 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:56:27,673 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1426c97[Count = 0] remaining members to acquire global barrier 2023-03-29 03:56:27,673 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:56:27,673 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:27,675 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:27,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:27,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:27,676 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:56:27,676 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:56:27,676 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:56:27,676 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:27,676 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:56:27,677 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:27,677 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:56:27,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:27,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:27,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:27,677 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:27,677 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:56:27,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:27,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:27,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:27,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:27,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:27,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:27,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:27,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:56:27,680 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:56:27,680 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:56:27,680 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:56:27,680 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:56:27,680 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:56:27,680 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:56:27,682 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:27,682 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:27,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:27,682 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:27,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:27,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:27,682 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:27,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:27,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:27,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:27,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:27,683 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:27,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:27,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:27,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:27,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:27,684 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:27,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:27,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:27,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:27,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:27,687 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:27,687 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:27,687 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:27,687 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:27,687 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:56:27,687 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:27,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:27,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:27,687 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:56:27,687 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:56:27,687 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:27,687 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:27,687 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:27,687 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:27,687 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:27,688 DEBUG [Thread-4303] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:56:27,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:27,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:27,688 DEBUG [Thread-4303] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:56:27,788 DEBUG [Thread-4303] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:56:27,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:56:27,790 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:56:27,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:56:27,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:56:27,791 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:56:27,791 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:27,791 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:56:27,791 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:56:27,792 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:27,792 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:27,794 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:27,794 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:27,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:27,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:27,794 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:27,794 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:56:27,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:27,795 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:27,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:56:27,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:27,795 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:56:27,795 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:56:27,795 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:56:27,795 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:56:27,795 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:27,795 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:56:27,796 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:56:27,796 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:56:27,796 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:27,796 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. started... 2023-03-29 03:56:27,796 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 94b16ca160750195aa137348cdb80ba9: 2023-03-29 03:56:27,796 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:27,796 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:56:27,796 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:56:27,796 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:27,796 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:56:27,796 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:56:27,798 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:27,798 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:27,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:27,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:27,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:27,798 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:27,798 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:56:27,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:27,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:27,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:27,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:27,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:27,800 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:56:27,800 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f0ed6a2[Count = 0] remaining members to acquire global barrier 2023-03-29 03:56:27,800 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:56:27,800 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:27,801 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:27,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:27,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:27,801 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:56:27,801 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:56:27,801 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:56:27,801 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:27,801 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:56:27,802 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:27,802 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:56:27,802 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:27,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:27,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:27,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:27,803 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:56:27,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:27,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:27,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:27,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:27,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:27,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:27,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:27,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:56:27,805 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:56:27,805 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:56:27,805 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:56:27,805 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:56:27,805 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:56:27,805 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:56:27,807 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:27,807 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:27,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:27,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:27,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:27,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:27,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:27,808 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:27,808 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:27,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:27,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:27,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:27,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:27,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:27,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:27,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:27,809 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:27,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:27,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:27,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:27,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:27,812 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:27,812 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:27,812 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:56:27,812 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:27,812 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:27,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:27,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:27,812 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:27,812 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:56:27,812 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:56:27,812 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:27,812 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:27,812 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:27,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:27,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:27,813 DEBUG [Thread-4303] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:56:27,812 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:27,813 DEBUG [Thread-4303] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:56:27,813 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:27,913 DEBUG [Thread-4303] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:56:27,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:56:27,915 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:56:27,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:56:27,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:56:27,916 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:56:27,916 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:27,916 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:56:27,916 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:56:27,916 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:27,917 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:27,918 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:27,918 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:27,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:27,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:27,918 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:27,918 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:56:27,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:27,918 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:27,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:56:27,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:27,919 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:56:27,919 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:56:27,919 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:56:27,919 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:56:27,919 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:27,919 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:56:27,919 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:56:27,920 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:56:27,920 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:27,920 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. started... 2023-03-29 03:56:27,920 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 94b16ca160750195aa137348cdb80ba9: 2023-03-29 03:56:27,920 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:27,920 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:56:27,920 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:56:27,920 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:27,920 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:56:27,920 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:56:27,921 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:27,922 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:27,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:27,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:27,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:27,922 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:27,922 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:56:27,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:27,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:27,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:27,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:27,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:27,923 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:56:27,923 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@39fcc96d[Count = 0] remaining members to acquire global barrier 2023-03-29 03:56:27,923 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:56:27,923 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:27,928 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:27,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:27,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:27,928 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:56:27,928 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:27,928 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:56:27,928 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:56:27,928 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:56:27,930 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:27,930 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:56:27,930 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:27,930 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:56:27,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:27,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:27,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:27,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:27,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:27,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:27,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:27,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:27,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:27,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:27,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:56:27,933 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:56:27,933 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:56:27,933 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:56:27,933 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:56:27,933 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:56:27,933 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:56:27,934 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:27,934 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:27,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:27,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:27,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:27,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:27,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:27,934 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:27,935 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:27,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:27,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:27,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:27,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:27,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:27,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:27,936 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:27,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:27,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:27,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:27,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:27,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:27,938 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:27,939 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:27,938 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:27,939 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:27,939 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:27,939 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:56:27,939 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:56:27,939 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:27,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:27,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:27,939 DEBUG [Thread-4303] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:56:27,939 DEBUG [Thread-4303] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:56:27,939 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:56:27,939 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:27,939 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:27,939 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:27,939 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:27,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:27,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:28,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on 94b16ca160750195aa137348cdb80ba9 2023-03-29 03:56:28,011 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 94b16ca160750195aa137348cdb80ba9 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-29 03:56:28,012 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94b16ca160750195aa137348cdb80ba9, store=A 2023-03-29 03:56:28,012 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:56:28,012 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94b16ca160750195aa137348cdb80ba9, store=B 2023-03-29 03:56:28,012 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:56:28,012 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94b16ca160750195aa137348cdb80ba9, store=C 2023-03-29 03:56:28,012 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:56:28,029 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:28,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37670 deadline: 1680062248026, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:28,030 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:28,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37674 deadline: 1680062248027, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:28,030 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:28,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37712 deadline: 1680062248027, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:28,030 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:28,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37666 deadline: 1680062248028, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:28,030 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:28,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37652 deadline: 1680062248028, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:28,039 DEBUG [Thread-4303] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:56:28,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:56:28,041 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:56:28,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:56:28,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:56:28,043 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:56:28,043 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:28,043 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:56:28,043 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:56:28,043 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:28,044 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:28,046 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:28,046 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:28,046 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:56:28,047 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:28,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:28,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:28,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:28,048 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:28,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:56:28,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:28,048 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:56:28,049 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:56:28,049 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:56:28,049 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:56:28,049 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:28,049 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:56:28,049 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:56:28,049 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:56:28,049 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:28,049 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. started... 2023-03-29 03:56:28,049 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. as already flushing 2023-03-29 03:56:28,049 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:28,049 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:56:28,049 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:56:28,050 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:28,050 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:56:28,050 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:56:28,051 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:28,051 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:28,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:28,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:28,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:28,052 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:28,052 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:56:28,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:28,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:28,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:28,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:28,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:28,053 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:56:28,053 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@60d9da2d[Count = 0] remaining members to acquire global barrier 2023-03-29 03:56:28,053 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:56:28,053 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:28,054 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:28,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:28,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:28,055 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:28,055 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:56:28,055 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:56:28,055 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:56:28,055 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:56:28,056 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:28,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:28,056 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:56:28,056 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:28,056 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:56:28,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:28,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:28,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:28,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:28,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:28,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:28,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:28,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:28,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:28,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:56:28,059 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:56:28,059 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:56:28,059 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:56:28,059 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:56:28,059 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:56:28,059 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:56:28,061 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:28,061 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:28,061 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:28,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:28,061 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:28,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:28,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:28,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:28,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:28,062 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:28,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:28,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:28,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:28,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:28,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:28,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:28,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:28,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:28,065 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:28,065 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:28,065 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:28,065 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:28,065 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:28,066 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:28,065 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:56:28,066 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:56:28,065 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:28,066 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:28,066 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:28,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:28,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:28,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:28,066 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:28,066 DEBUG [Thread-4303] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:56:28,066 DEBUG [Thread-4303] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:56:28,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:28,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:28,067 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:56:28,131 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:28,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37670 deadline: 1680062248130, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:28,132 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:28,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37674 deadline: 1680062248131, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:28,132 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:28,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37712 deadline: 1680062248131, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:28,132 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:28,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37666 deadline: 1680062248131, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:28,133 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:28,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37652 deadline: 1680062248131, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:28,166 DEBUG [Thread-4303] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:56:28,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:56:28,168 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:56:28,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:56:28,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:56:28,170 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:56:28,170 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:28,170 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:56:28,170 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:56:28,170 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:28,170 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:28,171 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:28,171 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:28,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:28,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:28,172 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:28,172 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:56:28,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:28,172 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:28,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:56:28,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:28,172 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:56:28,173 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:56:28,173 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:56:28,173 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:56:28,173 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:28,173 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:56:28,173 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:56:28,173 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:56:28,173 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:28,173 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. started... 2023-03-29 03:56:28,173 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. as already flushing 2023-03-29 03:56:28,173 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:28,173 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:56:28,173 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:56:28,173 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:28,173 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:56:28,173 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:56:28,175 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:28,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:28,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:28,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:28,175 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:28,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:28,175 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:28,175 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:56:28,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:28,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:28,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:28,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:28,177 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:56:28,177 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@72ed2475[Count = 0] remaining members to acquire global barrier 2023-03-29 03:56:28,177 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:56:28,177 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:28,179 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:28,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:28,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:28,179 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:56:28,179 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:28,179 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:56:28,179 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:56:28,179 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:56:28,181 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:56:28,181 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:28,181 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:56:28,181 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:28,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:28,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:28,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:28,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:28,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:28,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:28,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:28,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:28,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:28,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:28,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:56:28,183 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:56:28,183 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:56:28,183 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:56:28,183 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:56:28,183 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:56:28,183 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:56:28,185 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:28,185 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:28,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:28,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:28,185 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:28,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:28,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:28,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:28,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:28,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:28,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:28,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:28,186 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:28,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:28,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:28,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:28,187 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:28,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:28,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:28,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:28,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:28,190 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:28,190 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:28,190 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:28,190 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:28,190 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:28,190 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:28,190 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:28,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:28,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:28,190 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:28,190 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:28,190 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:56:28,190 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:56:28,190 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:28,190 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:56:28,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:28,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:28,190 DEBUG [Thread-4303] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:56:28,190 DEBUG [Thread-4303] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:56:28,291 DEBUG [Thread-4303] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:56:28,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:56:28,293 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:56:28,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:56:28,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:56:28,294 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:56:28,294 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:28,294 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:56:28,294 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:56:28,295 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:28,295 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:28,297 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:28,297 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:28,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:28,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:28,297 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:28,297 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:56:28,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:28,297 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:28,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:56:28,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:28,298 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:56:28,298 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:56:28,298 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:56:28,298 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:56:28,298 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:28,298 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:56:28,298 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:56:28,298 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:56:28,299 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:28,299 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. started... 2023-03-29 03:56:28,299 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. as already flushing 2023-03-29 03:56:28,299 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:28,299 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:56:28,299 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:56:28,299 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:28,299 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:56:28,299 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:56:28,300 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:28,300 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:28,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:28,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:28,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:28,300 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:28,300 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:56:28,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:28,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:28,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:28,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:28,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:28,302 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:56:28,302 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b5f574c[Count = 0] remaining members to acquire global barrier 2023-03-29 03:56:28,302 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:56:28,302 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:28,304 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:28,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:28,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:28,304 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:56:28,304 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:56:28,304 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:56:28,308 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:56:28,308 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:28,308 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:28,308 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:56:28,309 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:56:28,309 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:56:28,309 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:56:28,309 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:56:28,309 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:56:28,309 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:56:28,309 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:56:28,310 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:28,310 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:28,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:28,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:28,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:28,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:28,310 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:28,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:28,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:28,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:28,311 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:28,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:28,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:28,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:28,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:28,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:28,313 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:28,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:28,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:28,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:28,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:28,315 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:28,315 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:28,315 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:28,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:28,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:28,315 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:28,315 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:28,315 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:28,315 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:28,315 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:28,315 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:28,315 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:28,315 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:56:28,315 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:56:28,315 DEBUG [Thread-4303] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:56:28,316 DEBUG [Thread-4303] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:56:28,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:28,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:28,316 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:56:28,336 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:28,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37674 deadline: 1680062248333, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:28,336 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:28,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37670 deadline: 1680062248333, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:28,336 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:28,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37652 deadline: 1680062248333, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:28,336 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:28,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37712 deadline: 1680062248334, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:28,337 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:28,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37666 deadline: 1680062248334, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:28,416 DEBUG [Thread-4303] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:56:28,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:56:28,418 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:56:28,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:56:28,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:56:28,419 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:56:28,419 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:28,419 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:56:28,419 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:56:28,420 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:28,420 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:28,421 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:28,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:28,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:28,421 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:28,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:28,422 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:28,422 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:28,422 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:56:28,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:56:28,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:28,423 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:56:28,423 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:56:28,423 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:56:28,423 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:56:28,423 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:28,423 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:56:28,424 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:56:28,424 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:56:28,424 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:28,424 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. started... 2023-03-29 03:56:28,424 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. as already flushing 2023-03-29 03:56:28,424 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:28,424 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:56:28,424 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:56:28,424 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:28,424 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:56:28,424 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:56:28,425 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230329a2ec5d0b939d4c43b300d3f2680d4fa4 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230329a2ec5d0b939d4c43b300d3f2680d4fa4 2023-03-29 03:56:28,427 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:28,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:28,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:28,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:28,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:28,428 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:28,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:28,428 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:28,428 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:56:28,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:28,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:28,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:28,429 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:56:28,429 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@24374eeb[Count = 0] remaining members to acquire global barrier 2023-03-29 03:56:28,429 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:56:28,430 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:28,431 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:28,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:28,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:28,431 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:56:28,431 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:56:28,431 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:56:28,432 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:28,432 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:56:28,432 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=397, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/A/57ac31b6f6154938a870c668c5c71262 2023-03-29 03:56:28,433 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:28,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:28,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:28,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:28,433 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:56:28,433 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:28,433 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:56:28,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:28,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:28,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:28,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:28,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:28,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:28,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:28,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:56:28,437 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:56:28,437 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:56:28,437 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:56:28,437 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:56:28,437 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:56:28,437 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:56:28,440 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:28,440 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:28,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:28,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:28,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:28,440 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:28,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:28,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:28,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:28,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:28,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:28,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:28,442 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:28,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:28,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:28,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:28,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:28,443 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:28,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:28,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:28,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:28,446 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:28,446 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:28,447 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:28,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:28,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:28,447 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:28,447 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:56:28,447 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:56:28,447 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:28,447 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:28,447 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:28,447 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:28,447 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:28,447 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:28,447 DEBUG [Thread-4303] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:56:28,447 DEBUG [Thread-4303] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:56:28,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:28,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:28,547 DEBUG [Thread-4303] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:56:28,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:56:28,549 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:56:28,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:56:28,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:56:28,551 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:56:28,551 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:28,551 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:56:28,551 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:56:28,551 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:28,551 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:28,553 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:28,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:28,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:28,553 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:28,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:28,553 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:28,553 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:56:28,553 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:28,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:56:28,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:28,554 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:56:28,554 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:56:28,554 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:56:28,554 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:56:28,554 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:28,554 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:56:28,554 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:56:28,554 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:56:28,554 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:28,554 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. started... 2023-03-29 03:56:28,555 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. as already flushing 2023-03-29 03:56:28,555 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:28,555 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:56:28,555 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:56:28,555 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:28,555 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:56:28,555 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:56:28,556 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:28,556 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:28,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:28,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:28,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:28,557 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:28,557 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:56:28,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:28,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:28,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:28,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:28,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:28,558 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:56:28,558 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a91e0f2[Count = 0] remaining members to acquire global barrier 2023-03-29 03:56:28,558 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:56:28,558 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:28,559 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:28,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:28,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:28,560 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:56:28,560 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:56:28,560 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:56:28,560 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:28,560 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:56:28,564 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:56:28,565 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:28,565 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:28,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:28,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:28,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:28,565 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:56:28,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:28,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:28,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:28,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:28,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:28,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:28,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:28,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:56:28,567 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:56:28,567 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:56:28,567 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:56:28,567 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:56:28,567 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:56:28,568 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:56:28,569 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:28,569 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:28,569 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:28,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:28,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:28,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:28,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:28,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:28,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:28,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:28,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:28,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:28,570 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:28,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:28,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:28,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:28,572 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:28,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:28,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:28,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:28,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:28,574 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:28,574 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:28,574 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:28,574 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:28,574 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:28,574 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:56:28,574 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:56:28,574 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:28,574 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:28,574 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:28,574 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:28,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:28,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:28,574 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:28,574 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:56:28,574 DEBUG [Thread-4303] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:56:28,575 DEBUG [Thread-4303] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:56:28,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:28,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:28,639 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:28,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37712 deadline: 1680062248637, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:28,639 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:28,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37670 deadline: 1680062248637, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:28,639 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:28,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37652 deadline: 1680062248638, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:28,639 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:28,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37674 deadline: 1680062248639, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:28,640 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:28,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37666 deadline: 1680062248639, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:28,675 DEBUG [Thread-4303] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:56:28,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:56:28,677 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:56:28,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:56:28,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:56:28,678 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:56:28,678 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:28,678 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:56:28,678 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:56:28,678 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:28,679 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:28,680 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:28,680 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:28,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:28,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:28,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:28,681 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:28,681 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:28,681 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:56:28,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:56:28,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:28,682 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:56:28,682 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:56:28,682 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:56:28,682 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:56:28,682 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:28,682 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:56:28,682 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:56:28,682 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:56:28,682 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:28,682 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. started... 2023-03-29 03:56:28,682 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. as already flushing 2023-03-29 03:56:28,682 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:28,682 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:56:28,682 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:56:28,683 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:28,683 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:56:28,683 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:56:28,686 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:28,686 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:28,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:28,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:28,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:28,686 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:28,686 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:56:28,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:28,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:28,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:28,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:28,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:28,688 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:56:28,688 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1144a359[Count = 0] remaining members to acquire global barrier 2023-03-29 03:56:28,688 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:56:28,688 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:28,689 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:28,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:28,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:28,689 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:56:28,689 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:56:28,689 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:56:28,690 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:28,690 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:56:28,691 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:56:28,691 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:28,691 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:28,691 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:56:28,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:28,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:28,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:28,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:28,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:28,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:28,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:28,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:28,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:28,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:28,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:56:28,694 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:56:28,694 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:56:28,694 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:56:28,694 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:56:28,694 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:56:28,694 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:56:28,696 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:28,696 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:28,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:28,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:28,696 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:28,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:28,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:28,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:28,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:28,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:28,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:28,697 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:28,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:28,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:28,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:28,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:28,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:28,699 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:28,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:28,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:28,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:28,705 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:28,705 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:28,705 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:28,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:28,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:28,705 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:28,705 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:56:28,705 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:56:28,705 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:28,705 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:28,706 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:28,706 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:28,706 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:28,706 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:28,706 DEBUG [Thread-4303] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:56:28,706 DEBUG [Thread-4303] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:56:28,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:28,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:28,806 DEBUG [Thread-4303] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:56:28,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:56:28,808 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:56:28,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:56:28,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:56:28,810 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:56:28,810 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:28,810 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:56:28,810 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:56:28,810 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:28,810 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:28,811 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:28,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:28,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:28,812 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:28,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:28,812 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:28,812 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:56:28,812 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:28,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:56:28,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:28,813 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:56:28,813 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:56:28,813 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:56:28,813 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:56:28,813 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:28,813 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:56:28,814 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:56:28,814 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:56:28,814 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:28,814 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. started... 2023-03-29 03:56:28,814 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. as already flushing 2023-03-29 03:56:28,814 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:28,814 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:56:28,814 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:56:28,814 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:28,814 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:56:28,814 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:56:28,815 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:28,815 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:28,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:28,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:28,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:28,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:28,816 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:28,817 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:56:28,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:28,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:28,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:28,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:28,818 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:56:28,818 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1020730a[Count = 0] remaining members to acquire global barrier 2023-03-29 03:56:28,818 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:56:28,818 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:28,819 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:28,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:28,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:28,820 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:56:28,820 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:56:28,820 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:56:28,820 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:28,820 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:56:28,822 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:28,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:28,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:28,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:28,822 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:56:28,822 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:28,822 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:56:28,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:28,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:28,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:28,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:28,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:28,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:28,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:28,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:56:28,825 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:56:28,825 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:56:28,825 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:56:28,825 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:56:28,825 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:56:28,825 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:56:28,827 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:28,827 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:28,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:28,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:28,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:28,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:28,827 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:28,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:28,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:28,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:28,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:28,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:28,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:28,828 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:28,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:28,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:28,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:28,829 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:28,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:28,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:28,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:28,831 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:28,831 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:28,832 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:28,832 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:28,832 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:56:28,832 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:56:28,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:28,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:28,832 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:28,832 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:28,832 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:28,832 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:28,832 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:28,832 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:28,832 DEBUG [Thread-4303] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:56:28,832 DEBUG [Thread-4303] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:56:28,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:28,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:28,846 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=397 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/B/c17d39257d914de4aad2152a7a258cb6 2023-03-29 03:56:28,857 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=397 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/C/b1d9e9da33bb48e29b1368ca97f2d542 2023-03-29 03:56:28,861 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/A/57ac31b6f6154938a870c668c5c71262 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/57ac31b6f6154938a870c668c5c71262 2023-03-29 03:56:28,865 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/57ac31b6f6154938a870c668c5c71262, entries=450, sequenceid=397, filesize=66.7 K 2023-03-29 03:56:28,866 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/B/c17d39257d914de4aad2152a7a258cb6 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/c17d39257d914de4aad2152a7a258cb6 2023-03-29 03:56:28,869 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/c17d39257d914de4aad2152a7a258cb6, entries=150, sequenceid=397, filesize=11.9 K 2023-03-29 03:56:28,870 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/C/b1d9e9da33bb48e29b1368ca97f2d542 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/b1d9e9da33bb48e29b1368ca97f2d542 2023-03-29 03:56:28,873 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/b1d9e9da33bb48e29b1368ca97f2d542, entries=150, sequenceid=397, filesize=11.9 K 2023-03-29 03:56:28,873 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 94b16ca160750195aa137348cdb80ba9 in 862ms, sequenceid=397, compaction requested=true 2023-03-29 03:56:28,873 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 94b16ca160750195aa137348cdb80ba9: 2023-03-29 03:56:28,873 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:56:28,873 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:56:28,873 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-29 03:56:28,873 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-29 03:56:28,873 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:56:28,874 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 114020 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-29 03:56:28,875 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1912): 94b16ca160750195aa137348cdb80ba9/A is initiating minor compaction (all files) 2023-03-29 03:56:28,875 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 46914 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-29 03:56:28,875 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 94b16ca160750195aa137348cdb80ba9/A in TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:28,875 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1912): 94b16ca160750195aa137348cdb80ba9/B is initiating minor compaction (all files) 2023-03-29 03:56:28,875 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/51570dbb8b0d45fa87eb33fd8e50f2ac, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/ce2387e240424455b65afa1be80c1acf, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/57ac31b6f6154938a870c668c5c71262] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp, totalSize=111.3 K 2023-03-29 03:56:28,875 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 94b16ca160750195aa137348cdb80ba9/B in TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:28,875 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/08961dda88994f33b09ce83318cf63c0, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/6c3d3f961160426aa3de2574eb4d22d6, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/0993640b669a4f09ae3a0cabfdc08c06, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/c17d39257d914de4aad2152a7a258cb6] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp, totalSize=45.8 K 2023-03-29 03:56:28,876 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 51570dbb8b0d45fa87eb33fd8e50f2ac, keycount=150, bloomtype=ROW, size=26.1 K, encoding=NONE, compression=NONE, seqNum=364, earliestPutTs=1680062185694 2023-03-29 03:56:28,876 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 08961dda88994f33b09ce83318cf63c0, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=346, earliestPutTs=1680062185579 2023-03-29 03:56:28,876 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting ce2387e240424455b65afa1be80c1acf, keycount=100, bloomtype=ROW, size=18.6 K, encoding=NONE, compression=NONE, seqNum=383, earliestPutTs=1680062185865 2023-03-29 03:56:28,876 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 6c3d3f961160426aa3de2574eb4d22d6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=364, earliestPutTs=1680062185694 2023-03-29 03:56:28,876 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 57ac31b6f6154938a870c668c5c71262, keycount=450, bloomtype=ROW, size=66.7 K, encoding=NONE, compression=NONE, seqNum=397, earliestPutTs=1680062188003 2023-03-29 03:56:28,876 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 0993640b669a4f09ae3a0cabfdc08c06, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=383, earliestPutTs=1680062185865 2023-03-29 03:56:28,877 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting c17d39257d914de4aad2152a7a258cb6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=397, earliestPutTs=1680062188010 2023-03-29 03:56:28,885 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] throttle.PressureAwareThroughputController(145): 94b16ca160750195aa137348cdb80ba9#A#compaction#655 average throughput is 6.57 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:56:28,885 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] throttle.PressureAwareThroughputController(145): 94b16ca160750195aa137348cdb80ba9#B#compaction#656 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:56:28,904 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/B/378a1f6a9b534948be4de07e90e13e3f as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/378a1f6a9b534948be4de07e90e13e3f 2023-03-29 03:56:28,909 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 94b16ca160750195aa137348cdb80ba9/B of 94b16ca160750195aa137348cdb80ba9 into 378a1f6a9b534948be4de07e90e13e3f(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:56:28,910 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 94b16ca160750195aa137348cdb80ba9: 2023-03-29 03:56:28,910 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9., storeName=94b16ca160750195aa137348cdb80ba9/B, priority=12, startTime=1680062188873; duration=0sec 2023-03-29 03:56:28,910 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:56:28,910 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-29 03:56:28,912 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 46914 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-29 03:56:28,912 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1912): 94b16ca160750195aa137348cdb80ba9/C is initiating minor compaction (all files) 2023-03-29 03:56:28,912 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 94b16ca160750195aa137348cdb80ba9/C in TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:28,913 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/9997ca2063044d3b89fedb3f63e599c4, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/50a0006b926e457fa286a13cb3d0ff24, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/47647c64df1d43358d3ae25f251add7d, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/b1d9e9da33bb48e29b1368ca97f2d542] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp, totalSize=45.8 K 2023-03-29 03:56:28,913 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 9997ca2063044d3b89fedb3f63e599c4, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=346, earliestPutTs=1680062185579 2023-03-29 03:56:28,914 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 50a0006b926e457fa286a13cb3d0ff24, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=364, earliestPutTs=1680062185694 2023-03-29 03:56:28,914 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/A/09bfad8a80fc4065acbe5fab950215ee as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/09bfad8a80fc4065acbe5fab950215ee 2023-03-29 03:56:28,914 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 47647c64df1d43358d3ae25f251add7d, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=383, earliestPutTs=1680062185865 2023-03-29 03:56:28,915 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting b1d9e9da33bb48e29b1368ca97f2d542, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=397, earliestPutTs=1680062188010 2023-03-29 03:56:28,918 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 94b16ca160750195aa137348cdb80ba9/A of 94b16ca160750195aa137348cdb80ba9 into 09bfad8a80fc4065acbe5fab950215ee(size=26.2 K), total size for store is 26.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:56:28,918 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 94b16ca160750195aa137348cdb80ba9: 2023-03-29 03:56:28,919 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9., storeName=94b16ca160750195aa137348cdb80ba9/A, priority=13, startTime=1680062188873; duration=0sec 2023-03-29 03:56:28,919 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:56:28,922 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] throttle.PressureAwareThroughputController(145): 94b16ca160750195aa137348cdb80ba9#C#compaction#657 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:56:28,932 DEBUG [Thread-4303] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:56:28,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:56:28,934 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:56:28,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:56:28,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:56:28,936 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:56:28,936 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:28,936 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:56:28,936 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:56:28,936 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:28,936 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:28,938 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:28,938 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:28,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:28,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:28,939 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:28,939 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:56:28,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:28,940 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:28,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:56:28,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:28,940 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:56:28,941 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:56:28,941 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:56:28,941 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:56:28,941 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:28,941 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:56:28,941 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/C/ab929623b702435abdee9940aa0226df as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/ab929623b702435abdee9940aa0226df 2023-03-29 03:56:28,944 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:56:28,944 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:56:28,944 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:28,944 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. started... 2023-03-29 03:56:28,944 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 94b16ca160750195aa137348cdb80ba9 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-29 03:56:28,944 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94b16ca160750195aa137348cdb80ba9, store=A 2023-03-29 03:56:28,945 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:56:28,945 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94b16ca160750195aa137348cdb80ba9, store=B 2023-03-29 03:56:28,945 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:56:28,945 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94b16ca160750195aa137348cdb80ba9, store=C 2023-03-29 03:56:28,945 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:56:28,946 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 94b16ca160750195aa137348cdb80ba9/C of 94b16ca160750195aa137348cdb80ba9 into ab929623b702435abdee9940aa0226df(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:56:28,946 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 94b16ca160750195aa137348cdb80ba9: 2023-03-29 03:56:28,946 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9., storeName=94b16ca160750195aa137348cdb80ba9/C, priority=12, startTime=1680062188873; duration=0sec 2023-03-29 03:56:28,946 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:56:28,956 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230329920c94fb8f5543509b8cdd31b1dcef05 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230329920c94fb8f5543509b8cdd31b1dcef05 2023-03-29 03:56:28,962 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=419, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/A/6fcfc2d120014d98a52272fb58aa528e 2023-03-29 03:56:29,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on 94b16ca160750195aa137348cdb80ba9 2023-03-29 03:56:29,143 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. as already flushing 2023-03-29 03:56:29,151 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:29,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37670 deadline: 1680062249148, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:29,151 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:29,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37712 deadline: 1680062249149, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:29,151 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:29,151 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:29,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37652 deadline: 1680062249149, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:29,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37666 deadline: 1680062249150, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:29,157 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:29,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37674 deadline: 1680062249156, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:29,254 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:29,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37670 deadline: 1680062249252, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:29,254 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:29,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37712 deadline: 1680062249252, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:29,254 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:29,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37652 deadline: 1680062249252, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:29,254 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:29,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37666 deadline: 1680062249252, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:29,258 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:29,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37674 deadline: 1680062249258, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:29,373 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=419 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/B/97a7133da9404743a42ceeb0b60b5fb2 2023-03-29 03:56:29,456 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:29,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37670 deadline: 1680062249455, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:29,456 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:29,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37712 deadline: 1680062249456, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:29,457 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:29,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37652 deadline: 1680062249456, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:29,457 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:29,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37666 deadline: 1680062249456, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:29,460 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:29,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37674 deadline: 1680062249459, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:29,759 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:29,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37712 deadline: 1680062249757, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:29,759 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:29,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37670 deadline: 1680062249758, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:29,760 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:29,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37666 deadline: 1680062249758, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:29,760 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:29,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37652 deadline: 1680062249758, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:29,763 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:29,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37674 deadline: 1680062249762, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:29,783 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=419 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/C/dfb24adb4047429ebdbdc3626fec41b5 2023-03-29 03:56:29,787 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/A/6fcfc2d120014d98a52272fb58aa528e as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/6fcfc2d120014d98a52272fb58aa528e 2023-03-29 03:56:29,790 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/6fcfc2d120014d98a52272fb58aa528e, entries=150, sequenceid=419, filesize=25.4 K 2023-03-29 03:56:29,791 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/B/97a7133da9404743a42ceeb0b60b5fb2 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/97a7133da9404743a42ceeb0b60b5fb2 2023-03-29 03:56:29,797 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/97a7133da9404743a42ceeb0b60b5fb2, entries=150, sequenceid=419, filesize=11.9 K 2023-03-29 03:56:29,798 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/C/dfb24adb4047429ebdbdc3626fec41b5 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/dfb24adb4047429ebdbdc3626fec41b5 2023-03-29 03:56:29,802 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/dfb24adb4047429ebdbdc3626fec41b5, entries=150, sequenceid=419, filesize=11.9 K 2023-03-29 03:56:29,802 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=67.09 KB/68700 for 94b16ca160750195aa137348cdb80ba9 in 858ms, sequenceid=419, compaction requested=false 2023-03-29 03:56:29,803 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 94b16ca160750195aa137348cdb80ba9: 2023-03-29 03:56:29,803 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:29,803 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:56:29,803 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:56:29,803 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:29,804 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:56:29,804 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:56:29,806 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:29,806 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:29,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:29,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:29,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:29,807 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:29,807 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:56:29,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:29,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:29,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:29,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:29,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:29,809 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:56:29,809 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e7e37f3[Count = 0] remaining members to acquire global barrier 2023-03-29 03:56:29,809 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:56:29,809 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:29,811 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:29,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:29,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:29,811 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:56:29,811 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:56:29,811 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:56:29,811 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:29,811 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:56:29,812 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:56:29,812 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:29,812 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:56:29,813 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:29,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:29,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:29,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:29,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:29,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:29,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:29,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:29,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:29,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:29,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:29,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:56:29,817 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:56:29,817 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:56:29,817 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:56:29,817 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:56:29,817 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:56:29,817 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:56:29,819 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:29,819 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:29,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:29,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:29,819 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:29,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:29,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:29,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:29,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:29,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:29,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:29,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:29,820 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:29,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:29,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:29,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:29,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:29,821 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:29,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:29,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:29,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:29,825 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:29,825 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:29,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:29,825 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:29,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:29,825 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:29,825 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:56:29,825 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:56:29,825 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:29,825 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:29,825 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:29,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:29,825 DEBUG [Thread-4303] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:56:29,825 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:29,825 DEBUG [Thread-4303] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:56:29,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:29,825 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:29,825 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:29,925 DEBUG [Thread-4303] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:56:29,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:56:29,928 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:56:29,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:56:29,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:56:29,929 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:56:29,929 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:29,929 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:56:29,929 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:56:29,930 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:29,930 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:29,931 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:29,931 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:29,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:29,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:29,931 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:29,932 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:56:29,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:29,932 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:29,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:56:29,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:29,932 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:56:29,933 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:56:29,933 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:56:29,933 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:56:29,933 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:29,933 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:56:29,933 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:56:29,933 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:56:29,933 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:29,933 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. started... 2023-03-29 03:56:29,933 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 94b16ca160750195aa137348cdb80ba9 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-29 03:56:29,933 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94b16ca160750195aa137348cdb80ba9, store=A 2023-03-29 03:56:29,933 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:56:29,933 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94b16ca160750195aa137348cdb80ba9, store=B 2023-03-29 03:56:29,933 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:56:29,934 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94b16ca160750195aa137348cdb80ba9, store=C 2023-03-29 03:56:29,934 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:56:29,946 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303295b60a75f385a40e78fdaee0ff6c6d1bc to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303295b60a75f385a40e78fdaee0ff6c6d1bc 2023-03-29 03:56:29,951 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=433, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/A/dbadf17b79be41ce95bf89690cae8ce3 2023-03-29 03:56:29,960 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=433 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/B/19c6bd442a024904a38a99820d2ef985 2023-03-29 03:56:29,970 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=433 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/C/2360ffa733d54906bc0b03bfeb3e0245 2023-03-29 03:56:29,977 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/A/dbadf17b79be41ce95bf89690cae8ce3 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/dbadf17b79be41ce95bf89690cae8ce3 2023-03-29 03:56:29,980 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/dbadf17b79be41ce95bf89690cae8ce3, entries=150, sequenceid=433, filesize=25.4 K 2023-03-29 03:56:29,981 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/B/19c6bd442a024904a38a99820d2ef985 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/19c6bd442a024904a38a99820d2ef985 2023-03-29 03:56:29,984 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/19c6bd442a024904a38a99820d2ef985, entries=150, sequenceid=433, filesize=11.9 K 2023-03-29 03:56:29,985 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/C/2360ffa733d54906bc0b03bfeb3e0245 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/2360ffa733d54906bc0b03bfeb3e0245 2023-03-29 03:56:29,988 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/2360ffa733d54906bc0b03bfeb3e0245, entries=150, sequenceid=433, filesize=11.9 K 2023-03-29 03:56:29,989 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=0 B/0 for 94b16ca160750195aa137348cdb80ba9 in 56ms, sequenceid=433, compaction requested=true 2023-03-29 03:56:29,989 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 94b16ca160750195aa137348cdb80ba9: 2023-03-29 03:56:29,989 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:29,989 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:56:29,989 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:56:29,989 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:29,989 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:56:29,989 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:56:29,991 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:29,991 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:29,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:29,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:29,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:29,991 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:29,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:29,991 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:56:29,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:29,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:29,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:29,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:29,993 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:56:29,993 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b6dc0be[Count = 0] remaining members to acquire global barrier 2023-03-29 03:56:29,993 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:56:29,993 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:29,994 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:29,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:29,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:29,994 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:56:29,994 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:56:29,994 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:56:29,994 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:29,994 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:56:29,999 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:29,999 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:56:29,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:29,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:29,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:29,999 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:29,999 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:56:29,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:30,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:30,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:30,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:30,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:30,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:56:30,002 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:56:30,002 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:56:30,002 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:56:30,002 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:56:30,002 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:56:30,002 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:56:30,003 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:30,003 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:30,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:30,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:30,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:30,003 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:30,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:30,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:30,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:30,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:30,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:30,004 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:30,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:30,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:30,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:30,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,005 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:30,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:30,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,009 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:30,009 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,009 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:30,009 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:30,010 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:56:30,010 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:56:30,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:30,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:30,009 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:30,010 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,010 DEBUG [Thread-4303] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:56:30,009 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:30,010 DEBUG [Thread-4303] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:56:30,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:30,010 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:30,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:30,010 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:30,010 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:30,110 DEBUG [Thread-4303] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:56:30,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:56:30,112 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:56:30,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:56:30,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:56:30,114 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:56:30,114 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:30,114 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:56:30,114 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:56:30,114 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:30,114 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:30,117 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:30,117 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:30,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:30,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:30,117 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,117 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:56:30,117 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:30,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:56:30,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:30,118 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:56:30,118 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:56:30,118 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:56:30,118 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:56:30,118 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:30,118 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:56:30,118 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:56:30,118 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:56:30,118 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:30,118 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. started... 2023-03-29 03:56:30,119 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 94b16ca160750195aa137348cdb80ba9: 2023-03-29 03:56:30,119 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:30,119 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:56:30,119 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:56:30,119 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,119 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:56:30,119 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:56:30,120 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,120 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:30,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:30,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:30,121 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:30,121 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:56:30,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:30,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:30,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:30,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:30,122 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:56:30,122 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3f07f4e1[Count = 0] remaining members to acquire global barrier 2023-03-29 03:56:30,123 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:56:30,123 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:30,124 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:30,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:30,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:30,124 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:56:30,124 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:56:30,124 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:56:30,124 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,124 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:56:30,125 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:56:30,125 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:30,126 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:56:30,125 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:30,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:30,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:30,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:30,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:30,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:30,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:30,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:56:30,129 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:56:30,129 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:56:30,129 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:56:30,129 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:56:30,129 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:56:30,129 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:56:30,131 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:30,131 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:30,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:30,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:30,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:30,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:30,131 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:30,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:30,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:30,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:30,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:30,132 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:30,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:30,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:30,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:30,133 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:30,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:30,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,135 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:30,136 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:30,136 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,136 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:30,136 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:56:30,136 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:30,136 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:56:30,136 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:56:30,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:30,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:30,136 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:30,136 DEBUG [Thread-4303] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:56:30,136 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,136 DEBUG [Thread-4303] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:56:30,136 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:30,136 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:30,136 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:30,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:30,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:30,236 DEBUG [Thread-4303] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:56:30,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:56:30,238 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:56:30,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:56:30,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:56:30,240 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:56:30,240 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:30,240 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:56:30,240 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:56:30,240 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:30,240 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:30,241 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:30,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:30,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:30,241 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:30,242 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,242 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:56:30,242 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:30,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:56:30,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:30,242 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:56:30,243 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:56:30,243 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:56:30,243 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:56:30,243 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:30,243 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:56:30,244 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:56:30,244 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:56:30,244 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:30,244 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. started... 2023-03-29 03:56:30,244 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 94b16ca160750195aa137348cdb80ba9: 2023-03-29 03:56:30,245 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:30,245 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:56:30,245 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:56:30,245 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,245 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:56:30,245 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:56:30,246 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:30,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:30,247 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:30,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:30,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:30,247 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:30,247 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:56:30,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:30,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:30,248 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:56:30,248 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c2c6828[Count = 0] remaining members to acquire global barrier 2023-03-29 03:56:30,249 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:56:30,249 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:30,251 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:30,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:30,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:30,251 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:56:30,251 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:56:30,251 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:56:30,251 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,251 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:56:30,252 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:56:30,252 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,252 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:30,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:30,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:30,252 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:56:30,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:30,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:30,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:30,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:30,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:30,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:56:30,255 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:56:30,255 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:56:30,255 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:56:30,255 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:56:30,255 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:56:30,255 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:56:30,257 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:30,257 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:30,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:30,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:30,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:30,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:30,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:30,257 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:30,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:30,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:30,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:30,258 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:30,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:30,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:30,259 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:30,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:30,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:30,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,261 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,261 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:30,261 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:30,261 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,261 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:30,261 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:30,261 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:30,261 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:56:30,262 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:56:30,261 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:30,262 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:30,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:30,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:30,261 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:30,262 DEBUG [Thread-4303] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:56:30,262 DEBUG [Thread-4303] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:56:30,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:30,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:30,263 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:56:30,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on 94b16ca160750195aa137348cdb80ba9 2023-03-29 03:56:30,288 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 94b16ca160750195aa137348cdb80ba9 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-29 03:56:30,289 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94b16ca160750195aa137348cdb80ba9, store=A 2023-03-29 03:56:30,290 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:56:30,290 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94b16ca160750195aa137348cdb80ba9, store=B 2023-03-29 03:56:30,290 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:56:30,290 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94b16ca160750195aa137348cdb80ba9, store=C 2023-03-29 03:56:30,290 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:56:30,315 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:30,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37712 deadline: 1680062250310, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,315 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:30,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37666 deadline: 1680062250310, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,316 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:30,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37674 deadline: 1680062250311, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,316 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680062190309, max timestamp=1680062190314Num uniques 859; 2023-03-29 03:56:30,316 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680062190309, max timestamp=1680062190314Num uniques 859; 2023-03-29 03:56:30,317 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-29 03:56:30,317 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-29 03:56:30,318 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680062190309, max timestamp=1680062190314Num uniques 859; 2023-03-29 03:56:30,318 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-29 03:56:30,362 DEBUG [Thread-4303] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:56:30,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:56:30,364 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:56:30,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:56:30,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:56:30,366 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:56:30,366 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:30,366 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:56:30,366 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:56:30,366 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:30,366 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:30,368 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:30,368 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:30,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:30,369 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,369 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:56:30,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:30,369 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:30,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:56:30,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:30,370 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:56:30,370 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:56:30,370 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:56:30,370 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:56:30,370 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:30,370 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:56:30,371 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:56:30,371 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:56:30,371 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:30,371 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. started... 2023-03-29 03:56:30,371 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. as already flushing 2023-03-29 03:56:30,371 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:30,371 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:56:30,371 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:56:30,371 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,371 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:56:30,371 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:56:30,373 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:30,373 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:30,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:30,373 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:30,373 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:56:30,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:30,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:30,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:30,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:30,375 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:56:30,375 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@539213f1[Count = 0] remaining members to acquire global barrier 2023-03-29 03:56:30,375 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:56:30,375 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:30,376 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:30,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:30,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:30,376 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:56:30,376 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:56:30,376 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:56:30,376 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,376 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:56:30,377 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:30,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37670 deadline: 1680062250375, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,378 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:56:30,378 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,378 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:30,378 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:56:30,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:30,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:30,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:30,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:30,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:30,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:30,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:30,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,381 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:30,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:56:30,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37652 deadline: 1680062250378, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,381 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:56:30,381 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:56:30,381 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:56:30,381 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:56:30,381 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:56:30,381 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:56:30,385 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:30,385 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:30,385 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:30,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:30,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:30,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:30,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:30,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:30,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:30,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:30,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:30,386 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:30,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:30,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:30,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:30,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,388 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:30,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:30,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,390 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,390 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:30,390 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:30,390 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:30,390 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:30,390 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:56:30,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:30,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:30,390 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,390 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:30,390 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:30,390 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:56:30,390 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:30,390 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:30,391 DEBUG [Thread-4303] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:56:30,391 DEBUG [Thread-4303] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:56:30,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:30,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:30,418 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:30,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37712 deadline: 1680062250416, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,420 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:30,420 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:30,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37666 deadline: 1680062250416, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37674 deadline: 1680062250417, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,481 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:30,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37670 deadline: 1680062250479, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,484 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:30,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37652 deadline: 1680062250482, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,491 DEBUG [Thread-4303] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:56:30,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:56:30,493 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:56:30,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:56:30,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:56:30,494 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:56:30,494 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:30,494 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:56:30,494 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:56:30,494 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:30,494 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:30,498 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,498 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:30,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:30,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:30,499 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,499 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:56:30,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:30,499 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:30,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:56:30,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:30,500 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:56:30,500 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:56:30,500 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:56:30,500 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:56:30,500 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:30,500 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:56:30,500 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:56:30,500 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:56:30,500 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:30,500 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. started... 2023-03-29 03:56:30,501 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. as already flushing 2023-03-29 03:56:30,501 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:30,501 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:56:30,501 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:56:30,501 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,501 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:56:30,501 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:56:30,503 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,503 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:30,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:30,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:30,503 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:30,504 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:56:30,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:30,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:30,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:30,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:30,505 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:56:30,505 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@369f510d[Count = 0] remaining members to acquire global barrier 2023-03-29 03:56:30,505 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:56:30,505 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:30,506 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:30,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:30,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:30,507 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:56:30,507 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:56:30,507 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:56:30,507 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,507 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:56:30,508 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:56:30,508 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,508 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:30,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:30,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:30,508 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:56:30,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:30,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:30,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:30,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:30,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:30,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:56:30,511 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:56:30,511 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:56:30,511 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:56:30,511 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:56:30,511 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:56:30,511 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:56:30,513 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:30,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:30,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:30,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:30,513 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:30,513 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:30,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:30,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:30,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:30,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:30,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:30,513 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:30,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:30,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:30,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:30,515 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:30,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:30,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,518 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:30,518 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:30,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:30,518 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:30,518 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:56:30,518 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:56:30,518 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:30,519 DEBUG [Thread-4303] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:56:30,518 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:56:30,518 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:30,519 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:30,519 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,519 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:30,519 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:30,519 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:30,519 DEBUG [Thread-4303] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:56:30,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:30,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:30,619 DEBUG [Thread-4303] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:56:30,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:56:30,621 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:56:30,622 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:30,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37712 deadline: 1680062250620, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:56:30,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:56:30,622 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:56:30,622 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:30,622 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:56:30,622 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:56:30,623 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:30,623 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:30,624 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,624 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:30,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:30,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:30,625 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,625 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:56:30,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:30,625 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:30,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37666 deadline: 1680062250621, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,625 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:30,625 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:30,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37674 deadline: 1680062250622, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:56:30,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:30,626 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:56:30,626 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:56:30,626 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:56:30,626 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:56:30,626 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:30,626 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:56:30,626 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:56:30,626 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:56:30,626 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:30,626 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. started... 2023-03-29 03:56:30,626 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. as already flushing 2023-03-29 03:56:30,626 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:30,627 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:56:30,627 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:56:30,627 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,627 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:56:30,627 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:56:30,628 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,628 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:30,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:30,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:30,628 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:30,628 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:56:30,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:30,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:30,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:30,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:30,630 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:56:30,630 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@363f1bc3[Count = 0] remaining members to acquire global barrier 2023-03-29 03:56:30,630 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:56:30,630 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:30,631 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:30,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:30,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:30,631 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:56:30,631 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:56:30,631 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,631 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:56:30,631 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:56:30,633 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,633 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:56:30,633 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:30,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:30,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:30,634 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:56:30,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:30,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:30,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:30,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:30,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:30,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:56:30,636 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:56:30,636 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:56:30,636 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:56:30,636 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:56:30,636 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:56:30,636 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:56:30,637 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:30,637 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:30,637 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:30,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:30,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:30,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:30,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:30,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:30,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:30,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:30,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:30,638 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:30,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:30,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:30,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:30,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,639 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:30,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:30,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,641 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:30,641 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:56:30,642 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:56:30,641 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:56:30,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:30,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:30,641 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,642 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:30,642 DEBUG [Thread-4303] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:56:30,641 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:30,641 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:30,642 DEBUG [Thread-4303] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:56:30,642 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:30,642 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,642 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:30,642 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:30,642 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:30,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:30,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:30,684 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:30,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37670 deadline: 1680062250683, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,688 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:30,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37652 deadline: 1680062250686, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,702 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032914f6dcca863642bf9443f6d0bbc51c6e to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032914f6dcca863642bf9443f6d0bbc51c6e 2023-03-29 03:56:30,742 DEBUG [Thread-4303] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:56:30,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:56:30,745 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:56:30,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:56:30,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:56:30,746 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:56:30,746 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:30,746 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:56:30,746 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:56:30,746 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:30,747 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:30,748 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,748 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:30,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:30,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:30,748 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:30,748 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:56:30,749 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:30,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:56:30,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:30,749 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:56:30,749 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:56:30,749 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:56:30,749 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:56:30,749 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:30,749 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:56:30,750 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:56:30,750 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:56:30,750 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:30,750 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. started... 2023-03-29 03:56:30,750 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. as already flushing 2023-03-29 03:56:30,750 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:30,750 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:56:30,750 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:56:30,750 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,750 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:56:30,750 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:56:30,754 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:30,754 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:30,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:30,754 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:30,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:30,754 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:56:30,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:30,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:30,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:30,756 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:56:30,756 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@799d4804[Count = 0] remaining members to acquire global barrier 2023-03-29 03:56:30,756 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:56:30,756 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:30,757 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:30,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:30,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:30,757 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:56:30,757 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:56:30,758 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:56:30,758 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,758 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:56:30,759 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,759 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:56:30,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:30,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:30,759 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:30,759 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:56:30,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:30,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:30,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:30,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:30,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:30,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:56:30,764 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:56:30,764 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:56:30,764 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:56:30,764 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:56:30,764 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:56:30,764 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:56:30,766 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:30,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:30,766 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:30,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:30,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:30,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:30,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:30,766 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:30,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:30,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:30,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:30,767 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:30,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:30,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:30,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:30,768 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:30,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:30,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,774 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:30,774 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:30,774 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:30,774 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,774 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:56:30,775 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:56:30,774 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:30,775 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:30,775 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,775 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:30,775 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:30,775 DEBUG [Thread-4303] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:56:30,775 DEBUG [Thread-4303] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:56:30,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:30,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:30,775 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:30,775 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:56:30,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:30,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:30,875 DEBUG [Thread-4303] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:56:30,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:56:30,877 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:56:30,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:56:30,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:56:30,879 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:56:30,879 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:30,879 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:56:30,879 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:56:30,879 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:30,879 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:30,881 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:30,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:30,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:30,881 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:30,882 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,882 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:56:30,882 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:30,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:56:30,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:30,882 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:56:30,882 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:56:30,883 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:56:30,883 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:56:30,883 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:30,883 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:56:30,883 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:56:30,883 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:56:30,883 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:30,883 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. started... 2023-03-29 03:56:30,883 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. as already flushing 2023-03-29 03:56:30,883 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:30,883 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:56:30,883 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:56:30,883 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,883 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:56:30,883 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:56:30,887 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,887 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:30,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:30,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:30,888 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:30,888 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:56:30,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:30,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:30,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:30,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:30,889 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:56:30,889 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ad3d15d[Count = 0] remaining members to acquire global barrier 2023-03-29 03:56:30,889 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:56:30,890 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:30,891 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:30,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:30,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:30,892 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:56:30,892 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:56:30,892 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:56:30,892 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,892 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:56:30,893 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:56:30,893 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:30,893 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,893 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:56:30,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:30,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:30,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:30,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:30,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:30,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:30,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:30,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:56:30,896 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:56:30,896 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:56:30,896 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:56:30,896 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:56:30,896 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:56:30,896 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:56:30,897 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:30,898 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:30,898 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:30,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:30,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:30,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:30,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:30,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:30,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:30,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:30,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:30,898 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:30,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:30,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:30,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:30,899 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:30,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:30,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,901 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,901 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:30,901 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:30,901 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:56:30,902 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:56:30,902 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:56:30,901 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:30,901 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:30,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:30,901 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:30,902 DEBUG [Thread-4303] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:56:30,902 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:30,902 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:30,902 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:30,902 DEBUG [Thread-4303] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:56:30,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:30,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:30,902 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:30,924 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:30,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37712 deadline: 1680062250923, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,927 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:30,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37666 deadline: 1680062250926, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,929 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:30,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37674 deadline: 1680062250926, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,988 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:30,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37670 deadline: 1680062250985, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:30,993 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:30,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37652 deadline: 1680062250990, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:31,002 DEBUG [Thread-4303] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:56:31,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:56:31,005 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:56:31,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:56:31,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:56:31,006 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:56:31,006 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:31,006 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:56:31,006 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:56:31,007 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:31,007 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:31,009 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:31,009 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:31,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:31,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:31,009 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:31,009 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:56:31,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:31,010 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:31,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:56:31,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:31,010 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:56:31,010 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:56:31,010 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:56:31,010 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:56:31,011 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:31,011 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:56:31,011 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:56:31,011 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:56:31,011 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:31,011 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. started... 2023-03-29 03:56:31,011 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. as already flushing 2023-03-29 03:56:31,011 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:31,011 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:56:31,011 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:56:31,011 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:31,011 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:56:31,011 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:56:31,013 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:31,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:31,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:31,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:31,013 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:31,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:31,014 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:31,014 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:56:31,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:31,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:31,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:31,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:31,016 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:56:31,016 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7688f833[Count = 0] remaining members to acquire global barrier 2023-03-29 03:56:31,016 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:56:31,016 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:31,017 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:31,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:31,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:31,017 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:56:31,017 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:56:31,017 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:56:31,017 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:31,017 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:56:31,018 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:31,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:31,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:31,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:31,018 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:56:31,019 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:31,019 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:56:31,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:31,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:31,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:31,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:31,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:31,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:31,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:31,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:56:31,023 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:56:31,023 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:56:31,024 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:56:31,024 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:56:31,024 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:56:31,024 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:56:31,026 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:31,026 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:31,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:31,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:31,026 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:31,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:31,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:31,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:31,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:31,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:31,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:31,028 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:31,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:31,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:31,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:31,029 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:31,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:31,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:31,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:31,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:31,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:31,031 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:31,031 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:31,031 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:31,031 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:31,031 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:31,031 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:31,031 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:31,031 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:31,031 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:31,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:31,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:31,031 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:56:31,031 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:56:31,031 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:31,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:31,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:31,031 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:56:31,032 DEBUG [Thread-4303] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:56:31,032 DEBUG [Thread-4303] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:56:31,108 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=447, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/A/98fb35f39d80418eaf39b148fd06851e 2023-03-29 03:56:31,132 DEBUG [Thread-4303] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:56:31,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:56:31,134 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:56:31,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:56:31,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:56:31,135 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:56:31,135 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:31,135 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:56:31,135 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:56:31,135 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:31,135 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:31,136 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:31,136 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:31,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:31,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:31,137 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:31,137 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:56:31,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:31,137 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:31,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:56:31,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:31,138 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:56:31,138 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:56:31,138 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:56:31,138 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:56:31,138 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:31,138 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:56:31,138 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:56:31,138 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:56:31,138 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:31,138 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. started... 2023-03-29 03:56:31,138 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. as already flushing 2023-03-29 03:56:31,138 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:31,138 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:56:31,138 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:56:31,138 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:31,139 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:56:31,139 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:56:31,140 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:31,140 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:31,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:31,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:31,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:31,140 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:31,140 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:56:31,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:31,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:31,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:31,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:31,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:31,142 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:56:31,142 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@38271422[Count = 0] remaining members to acquire global barrier 2023-03-29 03:56:31,142 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:56:31,142 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:31,144 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:31,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:31,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:31,144 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:56:31,144 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:56:31,144 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:56:31,144 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:31,144 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:56:31,145 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:31,145 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:56:31,145 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:31,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:31,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:31,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:31,146 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:56:31,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:31,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:31,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:31,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:31,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:31,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:31,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:31,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:56:31,148 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:56:31,148 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:56:31,148 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:56:31,148 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:56:31,148 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:56:31,148 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:56:31,149 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:31,149 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:31,149 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:31,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:31,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:31,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:31,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:31,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:31,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:31,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:31,150 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:31,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:31,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:31,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:31,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:31,151 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:31,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:31,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:31,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:31,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:31,153 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:31,153 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:31,153 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:31,153 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:31,153 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:31,153 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:31,153 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:31,153 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:31,153 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:31,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:31,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:31,153 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:31,153 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:56:31,153 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:56:31,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:31,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:31,154 DEBUG [Thread-4303] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:56:31,154 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:56:31,154 DEBUG [Thread-4303] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:56:31,254 DEBUG [Thread-4303] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:56:31,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:56:31,256 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:56:31,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:56:31,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:56:31,257 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:56:31,257 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:31,257 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:56:31,257 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:56:31,258 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:31,258 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:31,260 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:31,260 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:31,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:31,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:31,261 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:31,261 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:56:31,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:31,261 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:31,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:56:31,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:31,262 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:56:31,262 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:56:31,262 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:56:31,262 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:56:31,262 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:31,262 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:56:31,262 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:56:31,262 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:56:31,262 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:31,262 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. started... 2023-03-29 03:56:31,262 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. as already flushing 2023-03-29 03:56:31,262 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:31,262 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:56:31,262 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:56:31,262 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:31,263 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:56:31,263 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:56:31,264 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:31,264 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:31,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:31,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:31,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:31,264 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:31,264 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:56:31,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:31,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:31,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:31,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:31,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:31,266 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:56:31,266 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b69a64a[Count = 0] remaining members to acquire global barrier 2023-03-29 03:56:31,266 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:56:31,266 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:31,267 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:31,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:31,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:31,268 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:56:31,268 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:56:31,268 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:56:31,268 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:31,268 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:56:31,273 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:31,273 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:56:31,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:31,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:31,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:31,273 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:31,273 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:56:31,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:31,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:31,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:31,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:31,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:31,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:31,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:31,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:56:31,275 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:56:31,275 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:56:31,275 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:56:31,275 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:56:31,275 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:56:31,275 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:56:31,277 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:31,277 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:31,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:31,277 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:31,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:31,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:31,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:31,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:31,277 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:31,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:31,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:31,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:31,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:31,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:31,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:31,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:31,280 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:31,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:31,283 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:31,283 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:31,283 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:31,283 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:31,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:31,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:31,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:31,283 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:31,283 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:31,283 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:31,283 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:31,283 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:31,283 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:31,283 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:56:31,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:31,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:31,284 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:56:31,284 DEBUG [Thread-4303] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:56:31,284 DEBUG [Thread-4303] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:56:31,284 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:56:31,384 DEBUG [Thread-4303] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:56:31,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:56:31,386 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:56:31,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:56:31,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:56:31,387 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:56:31,388 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:31,388 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:56:31,388 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:56:31,388 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:31,388 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:31,389 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:31,389 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:31,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:31,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:31,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:31,390 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:31,390 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:56:31,390 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:31,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:56:31,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:31,390 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:56:31,391 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:56:31,391 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:56:31,391 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:56:31,391 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:31,391 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:56:31,391 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:56:31,391 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:56:31,391 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:31,391 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. started... 2023-03-29 03:56:31,391 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. as already flushing 2023-03-29 03:56:31,391 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:31,391 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:56:31,391 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:56:31,391 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:31,391 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:56:31,391 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:56:31,394 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:31,394 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:31,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:31,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:31,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:31,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:31,394 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:31,394 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:56:31,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:31,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:31,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:31,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:31,396 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:56:31,396 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@f94d343[Count = 0] remaining members to acquire global barrier 2023-03-29 03:56:31,396 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:56:31,396 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:31,397 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:31,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:31,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:31,397 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:56:31,397 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:56:31,397 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:56:31,398 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:31,398 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:56:31,399 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:31,400 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:56:31,400 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:31,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:31,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:31,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:31,400 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:56:31,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:31,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:31,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:31,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:31,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:31,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:31,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:31,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:56:31,402 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:56:31,402 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:56:31,403 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:56:31,403 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:56:31,403 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:56:31,403 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:56:31,404 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:31,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:31,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:31,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:31,404 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:31,404 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:31,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:31,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:31,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:31,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:31,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:31,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:31,406 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:31,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:31,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:31,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:31,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:31,407 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:31,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:31,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:31,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:31,410 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:31,410 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:31,411 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:31,411 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:31,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:31,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:31,411 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:31,411 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:56:31,411 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:56:31,411 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:31,411 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:31,411 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:31,411 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:31,411 DEBUG [Thread-4303] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:56:31,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:31,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:31,411 DEBUG [Thread-4303] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:56:31,411 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:31,429 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:31,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37712 deadline: 1680062251428, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:31,433 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:31,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37666 deadline: 1680062251430, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:31,436 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:31,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37674 deadline: 1680062251434, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:31,492 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:31,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37670 deadline: 1680062251491, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:31,498 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:31,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37652 deadline: 1680062251496, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:31,511 DEBUG [Thread-4303] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:56:31,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:56:31,513 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:56:31,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:56:31,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:56:31,515 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:56:31,515 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:31,515 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:56:31,515 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:56:31,515 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:31,515 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:31,518 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:31,518 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:31,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:31,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:31,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:31,519 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:31,519 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:56:31,519 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:31,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:56:31,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:31,519 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:56:31,519 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:56:31,519 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:56:31,519 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:56:31,519 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:31,520 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:56:31,520 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:56:31,520 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:56:31,520 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:31,520 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. started... 2023-03-29 03:56:31,520 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. as already flushing 2023-03-29 03:56:31,520 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:31,520 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:56:31,520 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:56:31,520 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:31,520 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:56:31,520 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:56:31,522 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=447 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/B/4f9daa334e1d4ee59e584c0283d13ed7 2023-03-29 03:56:31,523 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:31,523 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:31,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:31,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:31,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:31,524 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:31,524 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:56:31,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:31,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:31,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:31,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:31,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:31,526 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:56:31,526 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3eecfbe2[Count = 0] remaining members to acquire global barrier 2023-03-29 03:56:31,526 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:56:31,526 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:31,527 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:31,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:31,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:31,527 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:56:31,527 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:56:31,527 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:56:31,528 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:31,528 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:56:31,529 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:56:31,529 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:31,529 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:56:31,529 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:31,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:31,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:31,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:31,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:31,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:31,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:31,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:31,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:31,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:31,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:31,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:56:31,532 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:56:31,532 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:56:31,532 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:56:31,532 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:56:31,532 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:56:31,532 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:56:31,532 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=447 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/C/8d6666c36f614d7f966944a505b1da12 2023-03-29 03:56:31,533 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:31,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:31,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:31,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:31,533 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:31,533 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:31,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:31,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:31,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:31,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:31,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:31,534 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:31,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:31,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:31,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:31,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:31,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:31,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:31,535 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:31,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:31,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:31,537 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/A/98fb35f39d80418eaf39b148fd06851e as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/98fb35f39d80418eaf39b148fd06851e 2023-03-29 03:56:31,539 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:31,539 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:31,539 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:31,539 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:31,539 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:31,539 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:31,539 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:31,539 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:31,539 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:31,539 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:56:31,539 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:56:31,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:31,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:31,539 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:31,539 DEBUG [Thread-4303] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:56:31,539 DEBUG [Thread-4303] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:56:31,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:31,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:31,541 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/98fb35f39d80418eaf39b148fd06851e, entries=450, sequenceid=447, filesize=66.7 K 2023-03-29 03:56:31,541 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/B/4f9daa334e1d4ee59e584c0283d13ed7 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/4f9daa334e1d4ee59e584c0283d13ed7 2023-03-29 03:56:31,544 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/4f9daa334e1d4ee59e584c0283d13ed7, entries=150, sequenceid=447, filesize=11.9 K 2023-03-29 03:56:31,545 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/C/8d6666c36f614d7f966944a505b1da12 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/8d6666c36f614d7f966944a505b1da12 2023-03-29 03:56:31,548 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/8d6666c36f614d7f966944a505b1da12, entries=150, sequenceid=447, filesize=11.9 K 2023-03-29 03:56:31,548 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=118.53 KB/121370 for 94b16ca160750195aa137348cdb80ba9 in 1260ms, sequenceid=447, compaction requested=true 2023-03-29 03:56:31,548 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 94b16ca160750195aa137348cdb80ba9: 2023-03-29 03:56:31,549 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:56:31,549 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-29 03:56:31,549 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-29 03:56:31,549 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:56:31,549 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:56:31,550 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 147216 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-29 03:56:31,550 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1912): 94b16ca160750195aa137348cdb80ba9/A is initiating minor compaction (all files) 2023-03-29 03:56:31,550 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49494 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-29 03:56:31,550 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 94b16ca160750195aa137348cdb80ba9/A in TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:31,550 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1912): 94b16ca160750195aa137348cdb80ba9/B is initiating minor compaction (all files) 2023-03-29 03:56:31,550 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/09bfad8a80fc4065acbe5fab950215ee, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/6fcfc2d120014d98a52272fb58aa528e, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/dbadf17b79be41ce95bf89690cae8ce3, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/98fb35f39d80418eaf39b148fd06851e] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp, totalSize=143.8 K 2023-03-29 03:56:31,550 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 94b16ca160750195aa137348cdb80ba9/B in TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:31,550 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/378a1f6a9b534948be4de07e90e13e3f, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/97a7133da9404743a42ceeb0b60b5fb2, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/19c6bd442a024904a38a99820d2ef985, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/4f9daa334e1d4ee59e584c0283d13ed7] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp, totalSize=48.3 K 2023-03-29 03:56:31,550 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 09bfad8a80fc4065acbe5fab950215ee, keycount=150, bloomtype=ROW, size=26.2 K, encoding=NONE, compression=NONE, seqNum=397, earliestPutTs=1680062188010 2023-03-29 03:56:31,550 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 378a1f6a9b534948be4de07e90e13e3f, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=397, earliestPutTs=1680062188010 2023-03-29 03:56:31,551 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 97a7133da9404743a42ceeb0b60b5fb2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=419, earliestPutTs=1680062188026 2023-03-29 03:56:31,551 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 6fcfc2d120014d98a52272fb58aa528e, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=419, earliestPutTs=1680062188026 2023-03-29 03:56:31,552 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 19c6bd442a024904a38a99820d2ef985, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=433, earliestPutTs=1680062189145 2023-03-29 03:56:31,552 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting dbadf17b79be41ce95bf89690cae8ce3, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=433, earliestPutTs=1680062189145 2023-03-29 03:56:31,552 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 4f9daa334e1d4ee59e584c0283d13ed7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=447, earliestPutTs=1680062190279 2023-03-29 03:56:31,552 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 98fb35f39d80418eaf39b148fd06851e, keycount=450, bloomtype=ROW, size=66.7 K, encoding=NONE, compression=NONE, seqNum=447, earliestPutTs=1680062190278 2023-03-29 03:56:31,562 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] throttle.PressureAwareThroughputController(145): 94b16ca160750195aa137348cdb80ba9#B#compaction#667 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:56:31,574 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] throttle.PressureAwareThroughputController(145): 94b16ca160750195aa137348cdb80ba9#A#compaction#668 average throughput is 1.41 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:56:31,594 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/B/7add7a28162f4da986732d342d667dd6 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/7add7a28162f4da986732d342d667dd6 2023-03-29 03:56:31,601 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 94b16ca160750195aa137348cdb80ba9/B of 94b16ca160750195aa137348cdb80ba9 into 7add7a28162f4da986732d342d667dd6(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:56:31,601 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 94b16ca160750195aa137348cdb80ba9: 2023-03-29 03:56:31,601 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9., storeName=94b16ca160750195aa137348cdb80ba9/B, priority=12, startTime=1680062191549; duration=0sec 2023-03-29 03:56:31,601 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:56:31,601 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-29 03:56:31,602 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49494 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-29 03:56:31,602 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1912): 94b16ca160750195aa137348cdb80ba9/C is initiating minor compaction (all files) 2023-03-29 03:56:31,602 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 94b16ca160750195aa137348cdb80ba9/C in TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:31,602 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/ab929623b702435abdee9940aa0226df, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/dfb24adb4047429ebdbdc3626fec41b5, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/2360ffa733d54906bc0b03bfeb3e0245, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/8d6666c36f614d7f966944a505b1da12] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp, totalSize=48.3 K 2023-03-29 03:56:31,603 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting ab929623b702435abdee9940aa0226df, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=397, earliestPutTs=1680062188010 2023-03-29 03:56:31,603 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting dfb24adb4047429ebdbdc3626fec41b5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=419, earliestPutTs=1680062188026 2023-03-29 03:56:31,604 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 2360ffa733d54906bc0b03bfeb3e0245, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=433, earliestPutTs=1680062189145 2023-03-29 03:56:31,604 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 8d6666c36f614d7f966944a505b1da12, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=447, earliestPutTs=1680062190279 2023-03-29 03:56:31,617 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/A/278e7ca258b84677bc787eef8a8b4db1 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/278e7ca258b84677bc787eef8a8b4db1 2023-03-29 03:56:31,631 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] throttle.PressureAwareThroughputController(145): 94b16ca160750195aa137348cdb80ba9#C#compaction#669 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:56:31,634 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 94b16ca160750195aa137348cdb80ba9/A of 94b16ca160750195aa137348cdb80ba9 into 278e7ca258b84677bc787eef8a8b4db1(size=26.3 K), total size for store is 26.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:56:31,634 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 94b16ca160750195aa137348cdb80ba9: 2023-03-29 03:56:31,635 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9., storeName=94b16ca160750195aa137348cdb80ba9/A, priority=12, startTime=1680062191548; duration=0sec 2023-03-29 03:56:31,635 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:56:31,640 DEBUG [Thread-4303] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:56:31,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:56:31,643 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:56:31,646 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/C/cfd96a66e048477db7c93f407268f4a7 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/cfd96a66e048477db7c93f407268f4a7 2023-03-29 03:56:31,651 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 94b16ca160750195aa137348cdb80ba9/C of 94b16ca160750195aa137348cdb80ba9 into cfd96a66e048477db7c93f407268f4a7(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:56:31,651 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 94b16ca160750195aa137348cdb80ba9: 2023-03-29 03:56:31,651 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9., storeName=94b16ca160750195aa137348cdb80ba9/C, priority=12, startTime=1680062191549; duration=0sec 2023-03-29 03:56:31,651 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:56:31,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:56:31,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:56:31,652 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:56:31,652 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:31,652 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:56:31,652 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:56:31,653 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:31,653 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:31,655 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:31,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:31,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:31,656 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:31,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:31,656 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:31,656 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:56:31,657 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:31,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:56:31,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:31,657 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:56:31,660 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:56:31,660 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:56:31,660 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:56:31,660 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:31,660 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:56:31,660 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:56:31,660 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:56:31,660 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:31,660 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. started... 2023-03-29 03:56:31,661 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 94b16ca160750195aa137348cdb80ba9 3/3 column families, dataSize=118.53 KB heapSize=335.38 KB 2023-03-29 03:56:31,661 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94b16ca160750195aa137348cdb80ba9, store=A 2023-03-29 03:56:31,661 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-29 03:56:31,661 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94b16ca160750195aa137348cdb80ba9, store=B 2023-03-29 03:56:31,661 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-29 03:56:31,661 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94b16ca160750195aa137348cdb80ba9, store=C 2023-03-29 03:56:31,661 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-29 03:56:32,089 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230329dc98d24f26d74e0abd2d90f69bbbb61c to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230329dc98d24f26d74e0abd2d90f69bbbb61c 2023-03-29 03:56:32,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on 94b16ca160750195aa137348cdb80ba9 2023-03-29 03:56:32,438 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. as already flushing 2023-03-29 03:56:32,452 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:32,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37712 deadline: 1680062252449, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:32,453 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:32,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37666 deadline: 1680062252450, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:32,453 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:32,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37674 deadline: 1680062252451, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:32,495 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=484, memsize=40.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/A/0f88f7db9db248a6951e27276f7e1c39 2023-03-29 03:56:32,502 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:32,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37670 deadline: 1680062252500, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:32,504 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:32,504 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=484 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/B/7255786c3ec0489482b9262ae610a3dd 2023-03-29 03:56:32,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37652 deadline: 1680062252503, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:32,519 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=484 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/C/1d1bc128614643e487e34b125b7b18a8 2023-03-29 03:56:32,522 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/A/0f88f7db9db248a6951e27276f7e1c39 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/0f88f7db9db248a6951e27276f7e1c39 2023-03-29 03:56:32,526 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/0f88f7db9db248a6951e27276f7e1c39, entries=150, sequenceid=484, filesize=25.4 K 2023-03-29 03:56:32,527 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/B/7255786c3ec0489482b9262ae610a3dd as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/7255786c3ec0489482b9262ae610a3dd 2023-03-29 03:56:32,531 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/7255786c3ec0489482b9262ae610a3dd, entries=150, sequenceid=484, filesize=11.9 K 2023-03-29 03:56:32,531 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/C/1d1bc128614643e487e34b125b7b18a8 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/1d1bc128614643e487e34b125b7b18a8 2023-03-29 03:56:32,535 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/1d1bc128614643e487e34b125b7b18a8, entries=150, sequenceid=484, filesize=11.9 K 2023-03-29 03:56:32,535 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~118.53 KB/121370, heapSize ~335.33 KB/343376, currentSize=60.38 KB/61830 for 94b16ca160750195aa137348cdb80ba9 in 874ms, sequenceid=484, compaction requested=false 2023-03-29 03:56:32,535 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 94b16ca160750195aa137348cdb80ba9: 2023-03-29 03:56:32,535 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:32,535 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:56:32,536 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:56:32,536 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:32,536 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:56:32,536 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:56:32,537 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:32,537 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:32,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:32,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:32,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:32,537 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:32,538 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:56:32,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:32,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:32,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:32,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:32,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:32,540 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:56:32,540 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e6a7e19[Count = 0] remaining members to acquire global barrier 2023-03-29 03:56:32,540 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:56:32,540 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:32,541 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:32,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:32,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:32,541 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:56:32,541 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:56:32,541 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:56:32,541 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:32,541 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:56:32,543 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:56:32,543 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:32,544 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:56:32,548 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:32,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:32,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:32,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:32,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:32,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:32,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:32,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:32,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:32,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:32,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:32,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:56:32,554 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:56:32,554 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:56:32,554 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:56:32,554 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:56:32,554 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:56:32,554 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:56:32,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on 94b16ca160750195aa137348cdb80ba9 2023-03-29 03:56:32,555 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 94b16ca160750195aa137348cdb80ba9 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-29 03:56:32,556 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94b16ca160750195aa137348cdb80ba9, store=A 2023-03-29 03:56:32,557 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:56:32,557 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94b16ca160750195aa137348cdb80ba9, store=B 2023-03-29 03:56:32,557 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:56:32,557 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94b16ca160750195aa137348cdb80ba9, store=C 2023-03-29 03:56:32,557 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:56:32,559 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:32,559 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:32,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:32,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:32,559 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:32,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:32,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:32,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:32,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:32,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:32,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:32,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:32,560 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:32,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:32,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:32,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:32,562 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:32,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:32,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:32,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:32,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:32,565 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:32,565 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:32,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:32,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:32,565 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:32,565 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:32,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:32,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:32,565 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:32,565 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:32,565 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:32,565 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:32,566 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:56:32,566 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:56:32,566 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:32,566 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:32,566 DEBUG [Thread-4303] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:56:32,566 DEBUG [Thread-4303] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:56:32,566 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:56:32,572 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230329d9796be8888b4f2ba1f3ad4ee620c2b3 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230329d9796be8888b4f2ba1f3ad4ee620c2b3 2023-03-29 03:56:32,577 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=498, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/A/08fa46538e654919906bb95a7b5793b3 2023-03-29 03:56:32,578 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:32,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37712 deadline: 1680062252576, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:32,581 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:32,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37674 deadline: 1680062252578, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:32,581 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:32,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37666 deadline: 1680062252578, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:32,666 DEBUG [Thread-4303] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:56:32,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:56:32,668 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:56:32,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:56:32,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:56:32,669 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:56:32,669 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:32,669 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:56:32,669 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:56:32,670 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:32,670 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:32,671 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:32,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:32,671 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:32,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:32,672 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:32,672 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:56:32,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:32,672 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:32,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:56:32,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:32,672 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:56:32,673 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:56:32,673 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:56:32,673 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:56:32,673 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:32,673 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:56:32,673 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:56:32,673 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:56:32,673 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:32,673 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. started... 2023-03-29 03:56:32,673 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. as already flushing 2023-03-29 03:56:32,673 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:32,673 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:56:32,673 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:56:32,673 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:32,673 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:56:32,673 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:56:32,676 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:32,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:32,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:32,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:32,676 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:32,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:32,676 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:32,676 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:56:32,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:32,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:32,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:32,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:32,677 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:56:32,677 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@58d7d468[Count = 0] remaining members to acquire global barrier 2023-03-29 03:56:32,677 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:56:32,677 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:32,678 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:32,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:32,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:32,679 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:56:32,679 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:56:32,679 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:56:32,679 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:32,679 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:56:32,680 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:32,680 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:56:32,680 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:32,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:32,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:32,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:32,680 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:56:32,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:32,682 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:32,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37712 deadline: 1680062252680, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:32,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:32,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:32,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:32,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:32,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:32,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:32,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:56:32,684 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:56:32,684 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:56:32,684 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:56:32,684 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:56:32,684 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:56:32,684 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:56:32,684 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:32,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37674 deadline: 1680062252682, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:32,685 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:32,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37666 deadline: 1680062252683, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:32,686 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:32,686 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:32,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:32,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:32,686 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:32,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:32,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:32,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:32,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:32,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:32,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:32,687 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:32,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:32,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:32,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:32,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:32,688 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:32,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:32,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:32,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:32,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:32,692 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:32,692 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:32,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:32,692 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:32,692 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:56:32,692 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:56:32,692 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:32,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:32,692 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:32,692 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:32,692 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:32,692 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:32,692 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:32,692 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:32,692 DEBUG [Thread-4303] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:56:32,692 DEBUG [Thread-4303] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:56:32,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:32,693 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:56:32,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:32,792 DEBUG [Thread-4303] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:56:32,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:56:32,794 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:56:32,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:56:32,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:56:32,795 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:56:32,796 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:32,796 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:56:32,796 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:56:32,796 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:32,796 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:32,797 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:32,797 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:32,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:32,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:32,797 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:32,798 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:56:32,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:32,798 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:32,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:56:32,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:32,798 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:56:32,798 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:56:32,799 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:56:32,799 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:56:32,799 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:32,799 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:56:32,799 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:56:32,799 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:56:32,799 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:32,799 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. started... 2023-03-29 03:56:32,799 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. as already flushing 2023-03-29 03:56:32,799 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:32,799 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:56:32,799 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:56:32,799 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:32,799 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:56:32,799 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:56:32,800 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:32,800 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:32,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:32,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:32,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:32,801 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:32,801 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:56:32,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:32,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:32,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:32,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:32,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:32,802 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:56:32,802 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@71700afa[Count = 0] remaining members to acquire global barrier 2023-03-29 03:56:32,802 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:56:32,802 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:32,803 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:32,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:32,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:32,804 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:56:32,804 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:56:32,804 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:56:32,804 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:32,804 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:56:32,807 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:32,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:32,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:32,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:32,807 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:56:32,807 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:32,807 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:56:32,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:32,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:32,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:32,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:32,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:32,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:32,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:32,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:56:32,809 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:56:32,809 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:56:32,809 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:56:32,809 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:56:32,809 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:56:32,809 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:56:32,811 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:32,811 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:32,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:32,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:32,811 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:32,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:32,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:32,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:32,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:32,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:32,812 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:32,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:32,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:32,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:32,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:32,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:32,813 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:32,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:32,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:32,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:32,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:32,815 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:32,815 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:32,815 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:32,815 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:56:32,815 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:56:32,815 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:32,815 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:32,815 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:32,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:32,815 DEBUG [Thread-4303] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:56:32,815 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:32,815 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:32,815 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:56:32,815 DEBUG [Thread-4303] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:56:32,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:32,815 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:32,815 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:32,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:32,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:32,885 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:32,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37712 deadline: 1680062252883, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:32,889 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:32,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37674 deadline: 1680062252886, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:32,889 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:32,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37666 deadline: 1680062252887, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:32,915 DEBUG [Thread-4303] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:56:32,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:56:32,917 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:56:32,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:56:32,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:56:32,918 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:56:32,919 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:32,919 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:56:32,919 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:56:32,919 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:32,919 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:32,920 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:32,920 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:32,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:32,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:32,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:32,921 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:32,921 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:56:32,921 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:32,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:56:32,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:32,921 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:56:32,922 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:56:32,922 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:56:32,922 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:56:32,922 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:32,922 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:56:32,922 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:56:32,922 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:56:32,922 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:32,922 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. started... 2023-03-29 03:56:32,922 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. as already flushing 2023-03-29 03:56:32,922 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:32,922 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:56:32,922 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:56:32,922 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:32,922 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:56:32,922 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:56:32,925 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:32,925 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:32,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:32,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:32,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:32,925 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:32,925 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:56:32,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:32,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:32,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:32,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:32,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:32,926 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:56:32,926 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@281adcdd[Count = 0] remaining members to acquire global barrier 2023-03-29 03:56:32,926 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:56:32,927 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:32,928 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:32,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:32,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:32,929 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:32,929 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:56:32,929 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:56:32,929 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:56:32,929 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:56:32,930 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:32,930 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:56:32,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:32,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:32,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:32,930 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:32,931 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:56:32,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:32,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:32,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:32,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:32,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:32,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:32,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:32,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:56:32,933 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:56:32,933 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:56:32,933 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:56:32,933 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:56:32,933 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:56:32,933 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:56:32,935 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:32,935 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:32,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:32,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:32,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:32,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:32,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:32,935 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:32,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:32,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:32,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:32,936 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:32,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:32,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:32,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:32,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:32,938 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:32,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:32,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:32,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:32,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:32,944 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:32,944 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:32,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:32,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:32,945 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:56:32,945 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:56:32,944 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:32,945 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:32,944 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:32,945 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:32,945 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:32,945 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:32,945 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:32,945 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:32,945 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:56:32,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:32,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:32,945 DEBUG [Thread-4303] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:56:32,945 DEBUG [Thread-4303] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:56:32,997 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=498 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/B/695af9f547f841a0bd277152911152e8 2023-03-29 03:56:33,011 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=498 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/C/92807488dc5f442ba8f7eb9bf9a76336 2023-03-29 03:56:33,015 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/A/08fa46538e654919906bb95a7b5793b3 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/08fa46538e654919906bb95a7b5793b3 2023-03-29 03:56:33,019 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/08fa46538e654919906bb95a7b5793b3, entries=150, sequenceid=498, filesize=25.4 K 2023-03-29 03:56:33,019 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/B/695af9f547f841a0bd277152911152e8 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/695af9f547f841a0bd277152911152e8 2023-03-29 03:56:33,022 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/695af9f547f841a0bd277152911152e8, entries=150, sequenceid=498, filesize=11.9 K 2023-03-29 03:56:33,023 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/C/92807488dc5f442ba8f7eb9bf9a76336 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/92807488dc5f442ba8f7eb9bf9a76336 2023-03-29 03:56:33,026 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/92807488dc5f442ba8f7eb9bf9a76336, entries=150, sequenceid=498, filesize=11.9 K 2023-03-29 03:56:33,027 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=107.34 KB/109920 for 94b16ca160750195aa137348cdb80ba9 in 472ms, sequenceid=498, compaction requested=true 2023-03-29 03:56:33,027 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 94b16ca160750195aa137348cdb80ba9: 2023-03-29 03:56:33,027 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:56:33,027 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-29 03:56:33,027 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-29 03:56:33,027 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:56:33,027 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:56:33,028 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 79058 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-29 03:56:33,028 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1912): 94b16ca160750195aa137348cdb80ba9/A is initiating minor compaction (all files) 2023-03-29 03:56:33,028 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 94b16ca160750195aa137348cdb80ba9/A in TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:33,028 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/278e7ca258b84677bc787eef8a8b4db1, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/0f88f7db9db248a6951e27276f7e1c39, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/08fa46538e654919906bb95a7b5793b3] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp, totalSize=77.2 K 2023-03-29 03:56:33,028 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37444 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-29 03:56:33,028 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1912): 94b16ca160750195aa137348cdb80ba9/B is initiating minor compaction (all files) 2023-03-29 03:56:33,028 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 94b16ca160750195aa137348cdb80ba9/B in TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:33,028 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/7add7a28162f4da986732d342d667dd6, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/7255786c3ec0489482b9262ae610a3dd, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/695af9f547f841a0bd277152911152e8] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp, totalSize=36.6 K 2023-03-29 03:56:33,028 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 278e7ca258b84677bc787eef8a8b4db1, keycount=150, bloomtype=ROW, size=26.3 K, encoding=NONE, compression=NONE, seqNum=447, earliestPutTs=1680062190279 2023-03-29 03:56:33,029 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 7add7a28162f4da986732d342d667dd6, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=447, earliestPutTs=1680062190279 2023-03-29 03:56:33,029 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 0f88f7db9db248a6951e27276f7e1c39, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=484, earliestPutTs=1680062190337 2023-03-29 03:56:33,029 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 7255786c3ec0489482b9262ae610a3dd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=484, earliestPutTs=1680062190337 2023-03-29 03:56:33,029 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 08fa46538e654919906bb95a7b5793b3, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=498, earliestPutTs=1680062192449 2023-03-29 03:56:33,029 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 695af9f547f841a0bd277152911152e8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=498, earliestPutTs=1680062192449 2023-03-29 03:56:33,035 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] throttle.PressureAwareThroughputController(145): 94b16ca160750195aa137348cdb80ba9#B#compaction#676 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:56:33,038 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] throttle.PressureAwareThroughputController(145): 94b16ca160750195aa137348cdb80ba9#A#compaction#677 average throughput is 4.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:56:33,045 DEBUG [Thread-4303] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:56:33,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:56:33,047 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:56:33,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:56:33,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:56:33,049 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:56:33,049 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:33,049 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:56:33,049 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:56:33,050 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:33,050 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:33,051 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:33,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:33,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:33,051 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:33,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:33,052 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:33,052 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:33,052 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:56:33,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:56:33,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:33,052 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:56:33,052 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:56:33,052 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:56:33,052 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:56:33,052 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:33,052 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:56:33,053 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:56:33,053 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:56:33,053 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:33,054 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. started... 2023-03-29 03:56:33,054 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 94b16ca160750195aa137348cdb80ba9 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-29 03:56:33,054 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94b16ca160750195aa137348cdb80ba9, store=A 2023-03-29 03:56:33,054 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:56:33,054 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94b16ca160750195aa137348cdb80ba9, store=B 2023-03-29 03:56:33,054 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:56:33,054 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94b16ca160750195aa137348cdb80ba9, store=C 2023-03-29 03:56:33,054 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:56:33,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on 94b16ca160750195aa137348cdb80ba9 2023-03-29 03:56:33,189 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. as already flushing 2023-03-29 03:56:33,201 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:33,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37712 deadline: 1680062253199, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:33,203 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:33,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37674 deadline: 1680062253200, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:33,203 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:33,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37666 deadline: 1680062253201, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:33,304 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:33,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37712 deadline: 1680062253302, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:33,304 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:33,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37674 deadline: 1680062253304, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:33,306 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:33,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37666 deadline: 1680062253304, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:33,446 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/B/91c27693585c4f9b81f20b4c3b8b326a as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/91c27693585c4f9b81f20b4c3b8b326a 2023-03-29 03:56:33,450 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 94b16ca160750195aa137348cdb80ba9/B of 94b16ca160750195aa137348cdb80ba9 into 91c27693585c4f9b81f20b4c3b8b326a(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:56:33,450 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 94b16ca160750195aa137348cdb80ba9: 2023-03-29 03:56:33,450 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9., storeName=94b16ca160750195aa137348cdb80ba9/B, priority=13, startTime=1680062193027; duration=0sec 2023-03-29 03:56:33,450 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:56:33,450 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-29 03:56:33,451 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37444 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-29 03:56:33,451 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1912): 94b16ca160750195aa137348cdb80ba9/C is initiating minor compaction (all files) 2023-03-29 03:56:33,451 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 94b16ca160750195aa137348cdb80ba9/C in TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:33,452 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/cfd96a66e048477db7c93f407268f4a7, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/1d1bc128614643e487e34b125b7b18a8, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/92807488dc5f442ba8f7eb9bf9a76336] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp, totalSize=36.6 K 2023-03-29 03:56:33,452 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting cfd96a66e048477db7c93f407268f4a7, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=447, earliestPutTs=1680062190279 2023-03-29 03:56:33,452 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 1d1bc128614643e487e34b125b7b18a8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=484, earliestPutTs=1680062190337 2023-03-29 03:56:33,453 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 92807488dc5f442ba8f7eb9bf9a76336, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=498, earliestPutTs=1680062192449 2023-03-29 03:56:33,459 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] throttle.PressureAwareThroughputController(145): 94b16ca160750195aa137348cdb80ba9#C#compaction#679 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:56:33,463 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/A/909e9c66618b4d10a32f16171ae8b859 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/909e9c66618b4d10a32f16171ae8b859 2023-03-29 03:56:33,467 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 94b16ca160750195aa137348cdb80ba9/A of 94b16ca160750195aa137348cdb80ba9 into 909e9c66618b4d10a32f16171ae8b859(size=26.4 K), total size for store is 26.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:56:33,468 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 94b16ca160750195aa137348cdb80ba9: 2023-03-29 03:56:33,468 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9., storeName=94b16ca160750195aa137348cdb80ba9/A, priority=13, startTime=1680062193027; duration=0sec 2023-03-29 03:56:33,468 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:56:33,468 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/C/93c93226f9fb473097488adce0ec58a4 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/93c93226f9fb473097488adce0ec58a4 2023-03-29 03:56:33,471 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 94b16ca160750195aa137348cdb80ba9/C of 94b16ca160750195aa137348cdb80ba9 into 93c93226f9fb473097488adce0ec58a4(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:56:33,472 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 94b16ca160750195aa137348cdb80ba9: 2023-03-29 03:56:33,472 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9., storeName=94b16ca160750195aa137348cdb80ba9/C, priority=13, startTime=1680062193027; duration=0sec 2023-03-29 03:56:33,472 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:56:33,474 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230329a9b828dc0cfb4ecc9d87b23b1ed6ca47 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230329a9b828dc0cfb4ecc9d87b23b1ed6ca47 2023-03-29 03:56:33,506 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:33,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37674 deadline: 1680062253505, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:33,507 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:33,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37712 deadline: 1680062253506, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:33,508 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:33,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37666 deadline: 1680062253507, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:33,809 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:33,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37674 deadline: 1680062253807, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:33,812 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:33,812 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:33,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37712 deadline: 1680062253810, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:33,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37666 deadline: 1680062253811, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:33,883 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=517, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/A/5d846473c6394ba2808bc0da9d78ba89 2023-03-29 03:56:33,895 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=517 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/B/ee8c886a3e4c41fa815e48a468cc625c 2023-03-29 03:56:33,904 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=517 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/C/93ad53cb25844d3ea023735a413b020a 2023-03-29 03:56:33,908 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/A/5d846473c6394ba2808bc0da9d78ba89 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/5d846473c6394ba2808bc0da9d78ba89 2023-03-29 03:56:33,913 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/5d846473c6394ba2808bc0da9d78ba89, entries=150, sequenceid=517, filesize=25.4 K 2023-03-29 03:56:33,914 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/B/ee8c886a3e4c41fa815e48a468cc625c as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/ee8c886a3e4c41fa815e48a468cc625c 2023-03-29 03:56:33,917 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/ee8c886a3e4c41fa815e48a468cc625c, entries=150, sequenceid=517, filesize=11.9 K 2023-03-29 03:56:33,918 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/C/93ad53cb25844d3ea023735a413b020a as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/93ad53cb25844d3ea023735a413b020a 2023-03-29 03:56:33,921 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/93ad53cb25844d3ea023735a413b020a, entries=150, sequenceid=517, filesize=11.9 K 2023-03-29 03:56:33,922 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 94b16ca160750195aa137348cdb80ba9 in 868ms, sequenceid=517, compaction requested=false 2023-03-29 03:56:33,922 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 94b16ca160750195aa137348cdb80ba9: 2023-03-29 03:56:33,922 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:33,922 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:56:33,922 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:56:33,922 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:33,922 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:56:33,922 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:56:33,923 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:33,923 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:33,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:33,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:33,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:33,924 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:33,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:33,924 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:56:33,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:33,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:33,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:33,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:33,926 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:56:33,926 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@45edf571[Count = 0] remaining members to acquire global barrier 2023-03-29 03:56:33,926 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:56:33,926 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:33,929 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:33,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:33,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:33,929 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:56:33,929 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:56:33,929 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:56:33,930 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:56:33,930 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:33,931 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:33,931 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:56:33,931 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:56:33,931 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:56:33,931 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:56:33,931 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:56:33,931 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:56:33,931 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:56:33,931 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:56:33,933 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:33,933 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:33,933 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:33,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:33,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:33,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:33,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:33,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:33,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:33,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:33,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:33,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:33,935 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:33,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:33,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:33,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:33,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:33,937 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:33,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:33,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:33,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:33,939 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:33,940 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:33,940 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:33,940 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:33,940 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:33,940 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:33,940 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:56:33,940 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:56:33,940 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:33,940 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:33,940 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:33,940 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:33,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:33,940 DEBUG [Thread-4303] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:56:33,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:33,940 DEBUG [Thread-4303] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:56:33,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:33,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:34,040 DEBUG [Thread-4303] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:56:34,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:56:34,043 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:56:34,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:56:34,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:56:34,044 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:56:34,044 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:34,044 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:56:34,044 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:56:34,045 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:34,045 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:34,046 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:34,046 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:34,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:34,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:34,047 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:34,047 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:56:34,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:34,047 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:34,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:56:34,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:34,048 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:56:34,048 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:56:34,048 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:56:34,048 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:56:34,048 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:34,048 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:56:34,048 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:56:34,048 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:56:34,049 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:34,049 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. started... 2023-03-29 03:56:34,049 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 94b16ca160750195aa137348cdb80ba9 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-29 03:56:34,049 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94b16ca160750195aa137348cdb80ba9, store=A 2023-03-29 03:56:34,049 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:56:34,049 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94b16ca160750195aa137348cdb80ba9, store=B 2023-03-29 03:56:34,049 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:56:34,049 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94b16ca160750195aa137348cdb80ba9, store=C 2023-03-29 03:56:34,049 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:56:34,066 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303290a70b3f16d4e4bcc995365818ef58b83 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303290a70b3f16d4e4bcc995365818ef58b83 2023-03-29 03:56:34,071 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=533, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/A/2e676da0a5854fc7b82830c3f7a54d45 2023-03-29 03:56:34,086 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=533 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/B/92b020720e494364add1a6fa86ceeb7f 2023-03-29 03:56:34,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on 94b16ca160750195aa137348cdb80ba9 2023-03-29 03:56:34,312 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. as already flushing 2023-03-29 03:56:34,329 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:34,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37666 deadline: 1680062254328, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:34,329 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:34,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37712 deadline: 1680062254329, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:34,330 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:34,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37674 deadline: 1680062254329, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:34,432 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:34,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37666 deadline: 1680062254430, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:34,432 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:34,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37712 deadline: 1680062254430, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:34,432 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:34,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37674 deadline: 1680062254431, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:34,495 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=533 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/C/a4671f180dc04808af5c7636ad4ea19c 2023-03-29 03:56:34,499 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/A/2e676da0a5854fc7b82830c3f7a54d45 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/2e676da0a5854fc7b82830c3f7a54d45 2023-03-29 03:56:34,503 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/2e676da0a5854fc7b82830c3f7a54d45, entries=150, sequenceid=533, filesize=25.4 K 2023-03-29 03:56:34,504 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/B/92b020720e494364add1a6fa86ceeb7f as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/92b020720e494364add1a6fa86ceeb7f 2023-03-29 03:56:34,507 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/92b020720e494364add1a6fa86ceeb7f, entries=150, sequenceid=533, filesize=11.9 K 2023-03-29 03:56:34,508 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:34,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37670 deadline: 1680062254505, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:34,510 DEBUG [Thread-4297] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4134 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9., hostname=jenkins-hbase4.apache.org,35599,1680062018534, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-29 03:56:34,511 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/C/a4671f180dc04808af5c7636ad4ea19c as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/a4671f180dc04808af5c7636ad4ea19c 2023-03-29 03:56:34,516 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:34,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37652 deadline: 1680062254515, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:34,518 DEBUG [Thread-4299] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4139 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9., hostname=jenkins-hbase4.apache.org,35599,1680062018534, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-29 03:56:34,519 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/a4671f180dc04808af5c7636ad4ea19c, entries=150, sequenceid=533, filesize=11.9 K 2023-03-29 03:56:34,520 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=114.05 KB/116790 for 94b16ca160750195aa137348cdb80ba9 in 471ms, sequenceid=533, compaction requested=true 2023-03-29 03:56:34,520 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 94b16ca160750195aa137348cdb80ba9: 2023-03-29 03:56:34,520 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:34,520 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:56:34,520 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:56:34,520 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:34,524 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:56:34,524 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:56:34,525 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:34,525 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:34,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:34,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:34,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:34,526 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:34,526 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:56:34,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:34,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:34,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:34,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:34,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:34,527 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:56:34,527 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@23ff1d30[Count = 0] remaining members to acquire global barrier 2023-03-29 03:56:34,527 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:56:34,527 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:34,528 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:34,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:34,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:34,528 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:56:34,528 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:56:34,528 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:56:34,529 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:34,529 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:56:34,530 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:34,530 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:56:34,530 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:34,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:34,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:34,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:34,530 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:56:34,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:34,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:34,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:34,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:34,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:34,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:34,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:34,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:56:34,532 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:56:34,532 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:56:34,532 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:56:34,532 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:56:34,532 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:56:34,532 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:56:34,535 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:34,535 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:34,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:34,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:34,535 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:34,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:34,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:34,536 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:34,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:34,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:34,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:34,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:34,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:34,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:34,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:34,537 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:34,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:34,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:34,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:34,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:34,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:34,539 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:34,539 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:34,539 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:34,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:34,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:34,539 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:34,539 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:56:34,540 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:56:34,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:34,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:34,540 DEBUG [Thread-4303] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:56:34,540 DEBUG [Thread-4303] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:56:34,539 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:34,540 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:34,540 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:34,540 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:34,540 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:34,540 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:34,540 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:56:34,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on 94b16ca160750195aa137348cdb80ba9 2023-03-29 03:56:34,636 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 94b16ca160750195aa137348cdb80ba9 3/3 column families, dataSize=120.76 KB heapSize=360.05 KB 2023-03-29 03:56:34,637 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680062194326, max timestamp=1680062194635Num uniques 859; 2023-03-29 03:56:34,637 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680062194326, max timestamp=1680062194635Num uniques 859; 2023-03-29 03:56:34,637 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-29 03:56:34,637 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-29 03:56:34,637 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680062194326, max timestamp=1680062194635Num uniques 859; 2023-03-29 03:56:34,637 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-29 03:56:34,638 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94b16ca160750195aa137348cdb80ba9, store=A 2023-03-29 03:56:34,638 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-29 03:56:34,638 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94b16ca160750195aa137348cdb80ba9, store=B 2023-03-29 03:56:34,638 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-29 03:56:34,638 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94b16ca160750195aa137348cdb80ba9, store=C 2023-03-29 03:56:34,638 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-29 03:56:34,640 DEBUG [Thread-4303] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:56:34,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:56:34,643 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:56:34,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:56:34,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:56:34,644 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:56:34,644 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:34,644 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:56:34,644 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:56:34,644 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:34,644 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:34,646 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:34,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:34,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:34,646 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:34,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:34,646 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:34,646 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:56:34,647 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:34,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:56:34,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:34,647 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:56:34,647 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:56:34,647 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:56:34,647 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:56:34,647 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:34,648 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:56:34,648 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:56:34,648 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:56:34,648 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:34,648 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. started... 2023-03-29 03:56:34,648 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. as already flushing 2023-03-29 03:56:34,648 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:34,648 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:56:34,648 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:56:34,648 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:34,648 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:56:34,648 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:56:34,650 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:34,650 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:34,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:34,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:34,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:34,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:34,651 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303299f730dc0ee3f4fbbb4cc51e2f705390a to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303299f730dc0ee3f4fbbb4cc51e2f705390a 2023-03-29 03:56:34,650 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:34,651 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:56:34,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:34,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:34,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:34,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:34,654 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:56:34,654 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5229e46f[Count = 0] remaining members to acquire global barrier 2023-03-29 03:56:34,654 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:56:34,655 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:34,657 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:34,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:34,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:34,657 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:34,657 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:56:34,657 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:56:34,657 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:56:34,657 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:56:34,659 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:34,659 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:56:34,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:34,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:34,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:34,659 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:34,659 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:56:34,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:34,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:34,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:34,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:34,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:34,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:34,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:34,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:56:34,663 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:56:34,663 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:56:34,663 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:56:34,663 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:56:34,663 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:56:34,663 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:56:34,665 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:34,665 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:34,665 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:34,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:34,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:34,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:34,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:34,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:34,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:34,666 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:34,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:34,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:34,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:34,668 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680062194651, max timestamp=1680062194665Num uniques 859; 2023-03-29 03:56:34,668 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680062194651, max timestamp=1680062194665Num uniques 859; 2023-03-29 03:56:34,668 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-29 03:56:34,668 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-29 03:56:34,668 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680062194651, max timestamp=1680062194665Num uniques 859; 2023-03-29 03:56:34,668 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-29 03:56:34,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:34,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:34,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:34,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:34,670 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:34,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:34,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:34,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:34,673 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:34,673 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:34,673 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:34,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:34,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:34,673 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:34,673 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:56:34,673 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:56:34,673 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:34,673 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:34,673 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:34,673 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:34,674 DEBUG [Thread-4303] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:56:34,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:34,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:34,674 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:34,674 DEBUG [Thread-4303] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:56:34,674 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:34,693 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1680062194685, max timestamp=1680062194690Num uniques 1068; 2023-03-29 03:56:34,693 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=12268, suffix heap size=139040,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=268 2023-03-29 03:56:34,693 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1680062194685, max timestamp=1680062194690Num uniques 1068; 2023-03-29 03:56:34,693 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=12268, suffix heap size=139040,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=268 2023-03-29 03:56:34,693 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1680062194685, max timestamp=1680062194690Num uniques 1068; 2023-03-29 03:56:34,693 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=12268, suffix heap size=139040,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=268 2023-03-29 03:56:34,724 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:34,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37666 deadline: 1680062254722, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:34,727 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:34,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37674 deadline: 1680062254724, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:34,727 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:34,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37712 deadline: 1680062254725, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:34,774 DEBUG [Thread-4303] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:56:34,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:56:34,776 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:56:34,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:56:34,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:56:34,777 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:56:34,777 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:34,777 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:56:34,777 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:56:34,778 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:34,778 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:34,783 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:34,783 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:34,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:34,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:34,783 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:34,783 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:56:34,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:34,783 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:34,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:56:34,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:34,784 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:56:34,784 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:56:34,784 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:56:34,784 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:34,784 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:56:34,785 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:56:34,785 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:56:34,785 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:34,785 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. started... 2023-03-29 03:56:34,785 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. as already flushing 2023-03-29 03:56:34,785 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:34,785 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:56:34,785 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:56:34,785 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:34,785 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:56:34,785 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:56:34,786 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:34,786 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:34,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:34,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:34,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:34,787 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:34,787 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:56:34,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:34,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:34,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:34,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:34,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:34,788 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:56:34,788 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c2b36c3[Count = 0] remaining members to acquire global barrier 2023-03-29 03:56:34,788 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:56:34,788 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:34,789 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:34,789 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:34,789 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:56:34,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:34,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:34,790 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:56:34,790 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:56:34,790 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:56:34,792 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:56:34,792 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:34,792 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:34,792 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:56:34,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:34,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:34,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:34,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:34,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:34,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:34,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:34,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:34,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:34,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:34,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:56:34,794 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:56:34,794 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:56:34,795 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:56:34,795 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:56:34,795 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:56:34,795 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:56:34,796 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:34,796 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:34,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:34,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:34,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:34,796 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:34,796 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:34,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:34,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:34,797 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:34,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:34,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:34,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:34,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:34,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:34,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:34,798 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:34,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:34,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:34,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:34,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:34,800 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:34,800 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:34,800 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:34,800 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:34,800 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:34,800 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:56:34,800 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:56:34,800 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:34,800 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:34,800 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:34,800 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:34,800 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:34,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:34,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:34,801 DEBUG [Thread-4303] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:56:34,801 DEBUG [Thread-4303] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:56:34,801 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:56:34,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:34,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:34,829 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:34,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37666 deadline: 1680062254826, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:34,830 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:34,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37674 deadline: 1680062254828, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:34,830 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:34,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37712 deadline: 1680062254828, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:34,901 DEBUG [Thread-4303] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:56:34,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:56:34,903 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:56:34,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:56:34,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:56:34,904 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:56:34,904 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:34,904 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:56:34,905 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:56:34,905 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:34,905 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:34,906 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:34,906 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:34,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:34,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:34,907 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:34,907 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:56:34,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:34,907 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:34,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:56:34,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:34,908 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:56:34,908 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:56:34,908 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:56:34,908 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:56:34,908 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:34,908 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:56:34,908 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:56:34,908 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:56:34,908 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:34,908 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. started... 2023-03-29 03:56:34,909 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. as already flushing 2023-03-29 03:56:34,909 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:34,909 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:56:34,909 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:56:34,909 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:34,909 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:56:34,909 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:56:34,911 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:34,912 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:34,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:34,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:34,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:34,912 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:34,912 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:56:34,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:34,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:34,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:34,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:34,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:34,913 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:56:34,914 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@647084a[Count = 0] remaining members to acquire global barrier 2023-03-29 03:56:34,914 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:56:34,914 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:34,915 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:34,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:34,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:34,915 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:56:34,915 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:56:34,915 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:56:34,915 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:34,915 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:56:34,916 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:56:34,916 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:34,916 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:34,916 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:56:34,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:34,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:34,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:34,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:34,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:34,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:34,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:34,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:34,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:34,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:34,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:56:34,919 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:56:34,919 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:56:34,919 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:56:34,919 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:56:34,919 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:56:34,919 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:56:34,921 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:34,921 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:34,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:34,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:34,921 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:34,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:34,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:34,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:34,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:34,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:34,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:34,922 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:34,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:34,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:34,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:34,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:34,923 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:34,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:34,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:34,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:34,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:34,926 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:34,926 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:34,926 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:34,926 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:34,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:34,926 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:56:34,926 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:56:34,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:34,926 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:56:34,926 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:34,927 DEBUG [Thread-4303] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:56:34,927 DEBUG [Thread-4303] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:56:34,926 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:34,927 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:34,927 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:34,927 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:34,926 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:34,927 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:34,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:35,027 DEBUG [Thread-4303] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:56:35,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:56:35,029 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:56:35,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:56:35,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:56:35,030 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:56:35,030 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:35,030 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:56:35,030 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:56:35,031 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:35,031 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:35,032 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:35,032 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:35,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:35,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:35,032 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:35,032 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:56:35,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:35,033 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:35,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:56:35,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:35,033 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:56:35,033 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:56:35,033 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:56:35,033 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:56:35,033 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:35,033 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:56:35,033 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:56:35,034 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:56:35,034 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:35,034 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. started... 2023-03-29 03:56:35,034 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. as already flushing 2023-03-29 03:56:35,034 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:35,034 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:56:35,034 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:56:35,034 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:35,034 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:56:35,034 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:56:35,034 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:35,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37666 deadline: 1680062255030, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:35,034 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:35,035 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:35,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37674 deadline: 1680062255031, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:35,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37712 deadline: 1680062255031, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:35,035 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:35,036 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:35,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:35,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:35,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:35,036 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:35,036 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:56:35,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:35,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:35,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:35,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:35,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:35,037 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:56:35,037 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5d8a832f[Count = 0] remaining members to acquire global barrier 2023-03-29 03:56:35,037 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:56:35,037 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:35,038 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:35,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:35,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:35,038 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:56:35,039 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:56:35,039 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:56:35,039 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:35,039 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:56:35,041 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:35,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:35,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:35,041 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:56:35,041 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:35,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:35,041 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:56:35,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:35,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:35,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:35,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:35,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:35,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:35,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:35,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:56:35,046 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:56:35,046 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:56:35,046 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:56:35,046 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:56:35,047 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:56:35,047 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:56:35,048 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:35,048 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:35,048 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:35,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:35,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:35,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:35,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:35,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:35,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:35,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:35,049 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:35,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:35,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:35,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:35,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:35,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:35,050 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:35,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:35,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:35,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:35,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:35,053 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:35,053 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:35,053 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:35,053 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:35,053 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:56:35,053 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:35,053 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:56:35,053 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:56:35,053 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:35,053 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:35,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:35,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:35,053 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:35,053 DEBUG [Thread-4303] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:56:35,053 DEBUG [Thread-4303] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:56:35,053 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:35,053 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:35,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:35,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:35,059 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=556, memsize=13.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/A/809da56654e443bd9cc70390e7feb12f 2023-03-29 03:56:35,153 DEBUG [Thread-4303] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:56:35,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:56:35,155 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:56:35,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:56:35,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:56:35,157 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:56:35,157 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:35,157 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:56:35,157 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:56:35,157 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:35,157 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:35,158 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:35,158 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:35,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:35,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:35,159 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:35,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:35,159 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:56:35,159 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:35,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:56:35,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:35,159 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:56:35,160 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:56:35,160 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:56:35,160 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:56:35,160 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:35,160 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:56:35,160 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:56:35,160 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:56:35,160 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:35,160 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. started... 2023-03-29 03:56:35,160 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. as already flushing 2023-03-29 03:56:35,160 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:35,160 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:56:35,160 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:56:35,160 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:35,160 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:56:35,161 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:56:35,165 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:35,165 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:35,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:35,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:35,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:35,165 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:35,166 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:56:35,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:35,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:35,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:35,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:35,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:35,167 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:56:35,167 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@401d0e6d[Count = 0] remaining members to acquire global barrier 2023-03-29 03:56:35,167 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:56:35,167 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:35,168 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:35,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:35,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:35,168 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:56:35,168 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:56:35,168 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:56:35,169 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:35,169 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:56:35,170 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:56:35,170 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:35,170 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:35,170 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:56:35,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:35,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:35,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:35,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:35,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:35,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:35,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:35,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:35,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:35,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:35,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:56:35,174 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:56:35,174 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:56:35,174 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:56:35,174 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:56:35,174 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:56:35,174 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:56:35,176 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:35,176 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:35,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:35,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:35,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:35,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:35,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:35,176 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:35,177 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:35,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:35,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:35,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:35,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:35,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:35,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:35,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:35,178 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:35,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:35,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:35,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:35,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:35,180 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:35,180 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:35,180 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:35,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:35,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:35,180 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:35,181 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:56:35,181 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:56:35,180 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:35,181 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:35,181 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:35,181 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:35,181 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:35,181 DEBUG [Thread-4303] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:56:35,181 DEBUG [Thread-4303] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:56:35,181 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:35,181 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:56:35,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:35,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:35,281 DEBUG [Thread-4303] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:56:35,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:56:35,283 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:56:35,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:56:35,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:56:35,284 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:56:35,284 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:35,284 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:56:35,284 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:56:35,285 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:35,285 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:35,286 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:35,286 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:35,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:35,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:35,287 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:35,287 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:56:35,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:35,287 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:35,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:56:35,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:35,287 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:56:35,288 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:56:35,288 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:56:35,288 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:56:35,288 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:35,288 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:56:35,288 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:56:35,288 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:56:35,288 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:35,288 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. started... 2023-03-29 03:56:35,288 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. as already flushing 2023-03-29 03:56:35,288 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:35,288 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:56:35,288 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:56:35,288 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:35,288 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:56:35,288 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:56:35,290 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:35,290 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:35,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:35,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:35,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:35,290 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:35,290 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:56:35,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:35,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:35,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:35,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:35,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:35,291 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:56:35,292 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@35f5b583[Count = 0] remaining members to acquire global barrier 2023-03-29 03:56:35,292 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:56:35,292 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:35,294 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:35,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:35,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:35,294 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:35,294 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:56:35,294 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:56:35,294 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:56:35,294 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:56:35,295 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:35,295 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:56:35,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:35,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:35,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:35,295 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:35,295 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:56:35,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:35,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:35,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:35,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:35,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:35,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:35,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:35,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:56:35,298 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:56:35,298 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:56:35,298 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:56:35,298 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:56:35,298 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:56:35,298 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:56:35,299 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:35,299 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:35,299 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:35,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:35,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:35,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:35,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:35,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:35,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:35,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:35,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:35,300 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:35,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:35,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:35,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:35,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:35,302 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:35,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:35,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:35,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:35,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:35,304 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:35,304 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:35,304 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:35,304 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:35,304 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:35,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:35,304 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:56:35,304 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:35,304 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:35,304 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:56:35,304 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:56:35,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:35,304 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:35,305 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:35,305 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:35,305 DEBUG [Thread-4303] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:56:35,305 DEBUG [Thread-4303] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:56:35,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:35,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:35,339 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:35,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37712 deadline: 1680062255337, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:35,339 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:35,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37674 deadline: 1680062255337, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:35,340 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:35,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37666 deadline: 1680062255337, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:35,405 DEBUG [Thread-4303] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:56:35,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:56:35,407 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:56:35,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:56:35,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:56:35,408 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:56:35,408 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:35,408 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:56:35,409 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:56:35,409 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:35,409 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:35,416 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:35,416 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:35,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:35,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:35,417 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:35,417 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:56:35,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:35,417 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:35,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:56:35,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:35,417 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:56:35,418 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:56:35,418 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:56:35,418 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:56:35,418 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:35,418 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:56:35,418 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:56:35,418 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:56:35,418 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:35,418 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. started... 2023-03-29 03:56:35,418 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. as already flushing 2023-03-29 03:56:35,418 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:35,418 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:56:35,418 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:56:35,418 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:35,418 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:56:35,418 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:56:35,420 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:35,420 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:35,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:35,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:35,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:35,420 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:35,420 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:56:35,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:35,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:35,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:35,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:35,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:35,422 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:56:35,422 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@bceee2f[Count = 0] remaining members to acquire global barrier 2023-03-29 03:56:35,422 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:56:35,422 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:35,423 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:35,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:35,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:35,423 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:56:35,423 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:56:35,423 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:56:35,423 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:35,423 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:56:35,424 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:35,424 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:56:35,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:35,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:35,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:35,424 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:35,424 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:56:35,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:35,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:35,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:35,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:35,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:35,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:35,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:35,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:56:35,427 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:56:35,427 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:56:35,427 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:56:35,427 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:56:35,427 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:56:35,427 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:56:35,429 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:35,429 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:35,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:35,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:35,429 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:35,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:35,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:35,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:35,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:35,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:35,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:35,430 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:35,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:35,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:35,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:35,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:35,431 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:35,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:35,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:35,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:35,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:35,433 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:35,433 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:35,433 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:35,433 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:35,433 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:35,433 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:56:35,433 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:56:35,433 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:35,433 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:35,433 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:56:35,433 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:35,433 DEBUG [Thread-4303] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:56:35,433 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:35,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:35,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:35,433 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:35,433 DEBUG [Thread-4303] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:56:35,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:35,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:35,468 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=556 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/B/747b3a2693804be7b44985c12b01b66d 2023-03-29 03:56:35,533 DEBUG [Thread-4303] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:56:35,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:56:35,535 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:56:35,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:56:35,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:56:35,537 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:56:35,537 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:35,537 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:56:35,537 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:56:35,537 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:35,537 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:35,538 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:35,538 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:35,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:35,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:35,539 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:35,539 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:56:35,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:35,539 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:35,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:56:35,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:35,539 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:56:35,540 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:56:35,540 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:56:35,540 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:56:35,540 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:35,540 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:56:35,540 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:56:35,540 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:56:35,540 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:35,540 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. started... 2023-03-29 03:56:35,540 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. as already flushing 2023-03-29 03:56:35,540 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:35,540 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:56:35,540 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:56:35,540 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:35,540 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:56:35,540 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:56:35,542 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:35,542 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:35,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:35,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:35,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:35,542 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:35,542 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:56:35,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:35,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:35,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:35,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:35,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:35,543 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:56:35,543 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d48782[Count = 0] remaining members to acquire global barrier 2023-03-29 03:56:35,543 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:56:35,544 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:35,546 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:35,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:35,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:35,546 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:56:35,546 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:35,546 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:56:35,546 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:56:35,546 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:56:35,549 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:56:35,549 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:35,549 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:35,549 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:56:35,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:35,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:35,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:35,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:35,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:35,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:35,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:35,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:35,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:35,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:35,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:56:35,551 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:56:35,551 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:56:35,551 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:56:35,551 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:56:35,551 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:56:35,551 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:56:35,553 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:35,553 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:35,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:35,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:35,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:35,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:35,553 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:35,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:35,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:35,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:35,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:35,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:35,554 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:35,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:35,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:35,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:35,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:35,555 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:35,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:35,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:35,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:35,557 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:35,557 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:35,557 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:35,557 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:56:35,557 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:56:35,557 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:35,557 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:35,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:35,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:35,558 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:35,557 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:56:35,557 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:35,558 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:35,558 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:35,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:35,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:35,558 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:35,558 DEBUG [Thread-4303] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:56:35,558 DEBUG [Thread-4303] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:56:35,658 DEBUG [Thread-4303] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:56:35,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:56:35,660 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:56:35,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:56:35,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:56:35,661 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:56:35,661 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:35,661 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:56:35,661 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:56:35,662 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:35,662 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:35,664 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:35,664 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:35,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:35,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:35,664 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:35,664 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:56:35,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:35,665 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:35,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:56:35,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:35,665 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:56:35,665 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:56:35,665 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:56:35,665 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:56:35,665 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:35,665 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:56:35,666 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:56:35,666 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:56:35,666 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:35,666 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. started... 2023-03-29 03:56:35,666 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. as already flushing 2023-03-29 03:56:35,666 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:35,666 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:56:35,666 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:56:35,666 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:35,666 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:56:35,666 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:56:35,667 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:35,667 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:35,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:35,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:35,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:35,667 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:35,667 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:56:35,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:35,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:35,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:35,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:35,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:35,669 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:56:35,669 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2a129fd3[Count = 0] remaining members to acquire global barrier 2023-03-29 03:56:35,669 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:56:35,669 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:35,670 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:35,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:35,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:35,671 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:56:35,671 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:56:35,671 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:56:35,671 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:35,671 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:56:35,672 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:35,672 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:56:35,672 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:35,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:35,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:35,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:35,672 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:56:35,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:35,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:35,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:35,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:35,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:35,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:35,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:35,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:56:35,674 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:56:35,674 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:56:35,674 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:56:35,674 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:56:35,674 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:56:35,674 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:56:35,676 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:35,676 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:35,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:35,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:35,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:35,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:35,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:35,676 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:35,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:35,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:35,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:35,677 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:35,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:35,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:35,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:35,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:35,678 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:35,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:35,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:35,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:35,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:35,680 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:35,680 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:35,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:35,681 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:56:35,681 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:56:35,681 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:35,681 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:35,681 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:35,681 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:35,681 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:35,681 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:35,681 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:35,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:35,681 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:35,681 DEBUG [Thread-4303] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:56:35,681 DEBUG [Thread-4303] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:56:35,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:35,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:35,682 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:56:35,781 DEBUG [Thread-4303] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:56:35,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:56:35,783 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:56:35,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:56:35,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:56:35,784 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:56:35,784 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:35,784 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:56:35,784 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:56:35,785 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:35,785 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:35,786 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:35,786 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:35,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:35,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:35,786 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:35,786 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:56:35,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:35,787 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:35,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:56:35,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:35,787 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:56:35,787 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:56:35,787 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:56:35,787 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:56:35,787 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:35,787 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:56:35,787 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:56:35,787 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:56:35,788 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:35,788 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. started... 2023-03-29 03:56:35,788 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. as already flushing 2023-03-29 03:56:35,788 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:35,788 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:56:35,788 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:56:35,788 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:35,788 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:56:35,788 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:56:35,789 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:35,789 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:35,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:35,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:35,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:35,790 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:35,790 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:56:35,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:35,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:35,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:35,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:35,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:35,792 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:56:35,792 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@32d36ebf[Count = 0] remaining members to acquire global barrier 2023-03-29 03:56:35,792 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:56:35,792 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:35,794 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:35,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:35,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:35,794 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:56:35,794 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:56:35,794 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:56:35,794 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:35,794 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:56:35,796 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:35,796 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:56:35,796 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:35,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:35,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:35,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:35,796 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:56:35,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:35,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:35,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:35,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:35,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:35,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:35,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:35,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:56:35,798 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:56:35,798 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:56:35,798 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:56:35,798 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:56:35,798 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:56:35,798 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:56:35,799 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:35,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:35,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:35,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:35,800 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:35,800 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:35,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:35,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:35,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:35,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:35,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:35,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:35,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:35,800 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:35,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:35,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:35,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:35,801 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:35,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:35,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:35,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:35,803 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:35,803 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:35,803 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:35,803 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:35,803 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:35,804 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:35,804 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:35,804 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:35,804 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:35,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:35,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:35,804 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:56:35,804 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:56:35,804 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:35,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:35,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:35,804 DEBUG [Thread-4303] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:56:35,804 DEBUG [Thread-4303] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:56:35,842 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:35,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37666 deadline: 1680062255840, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:35,842 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:35,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37712 deadline: 1680062255841, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:35,846 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:35,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37674 deadline: 1680062255843, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:35,881 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=556 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/C/713000b021b24cd4b0c15de1446c9c1a 2023-03-29 03:56:35,885 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/A/809da56654e443bd9cc70390e7feb12f as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/809da56654e443bd9cc70390e7feb12f 2023-03-29 03:56:35,888 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/809da56654e443bd9cc70390e7feb12f, entries=250, sequenceid=556, filesize=39.2 K 2023-03-29 03:56:35,889 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/B/747b3a2693804be7b44985c12b01b66d as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/747b3a2693804be7b44985c12b01b66d 2023-03-29 03:56:35,893 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/747b3a2693804be7b44985c12b01b66d, entries=150, sequenceid=556, filesize=11.9 K 2023-03-29 03:56:35,894 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/C/713000b021b24cd4b0c15de1446c9c1a as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/713000b021b24cd4b0c15de1446c9c1a 2023-03-29 03:56:35,897 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/713000b021b24cd4b0c15de1446c9c1a, entries=150, sequenceid=556, filesize=11.9 K 2023-03-29 03:56:35,898 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~106.03 KB/108576, currentSize=147.60 KB/151140 for 94b16ca160750195aa137348cdb80ba9 in 1262ms, sequenceid=556, compaction requested=true 2023-03-29 03:56:35,898 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 94b16ca160750195aa137348cdb80ba9: 2023-03-29 03:56:35,898 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-29 03:56:35,899 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:56:35,899 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-29 03:56:35,899 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:56:35,900 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:56:35,900 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 119290 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-29 03:56:35,900 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1912): 94b16ca160750195aa137348cdb80ba9/A is initiating minor compaction (all files) 2023-03-29 03:56:35,900 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 94b16ca160750195aa137348cdb80ba9/A in TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:35,900 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/909e9c66618b4d10a32f16171ae8b859, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/5d846473c6394ba2808bc0da9d78ba89, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/2e676da0a5854fc7b82830c3f7a54d45, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/809da56654e443bd9cc70390e7feb12f] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp, totalSize=116.5 K 2023-03-29 03:56:35,901 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49732 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-29 03:56:35,901 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 909e9c66618b4d10a32f16171ae8b859, keycount=150, bloomtype=ROW, size=26.4 K, encoding=NONE, compression=NONE, seqNum=498, earliestPutTs=1680062192449 2023-03-29 03:56:35,901 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1912): 94b16ca160750195aa137348cdb80ba9/B is initiating minor compaction (all files) 2023-03-29 03:56:35,901 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 94b16ca160750195aa137348cdb80ba9/B in TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:35,901 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/91c27693585c4f9b81f20b4c3b8b326a, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/ee8c886a3e4c41fa815e48a468cc625c, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/92b020720e494364add1a6fa86ceeb7f, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/747b3a2693804be7b44985c12b01b66d] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp, totalSize=48.6 K 2023-03-29 03:56:35,902 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 5d846473c6394ba2808bc0da9d78ba89, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=517, earliestPutTs=1680062192572 2023-03-29 03:56:35,902 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 2e676da0a5854fc7b82830c3f7a54d45, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=533, earliestPutTs=1680062193194 2023-03-29 03:56:35,903 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 91c27693585c4f9b81f20b4c3b8b326a, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=498, earliestPutTs=1680062192449 2023-03-29 03:56:35,903 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 809da56654e443bd9cc70390e7feb12f, keycount=250, bloomtype=ROW, size=39.2 K, encoding=NONE, compression=NONE, seqNum=556, earliestPutTs=1680062194327 2023-03-29 03:56:35,903 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting ee8c886a3e4c41fa815e48a468cc625c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=517, earliestPutTs=1680062192572 2023-03-29 03:56:35,903 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 92b020720e494364add1a6fa86ceeb7f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=533, earliestPutTs=1680062193194 2023-03-29 03:56:35,904 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 747b3a2693804be7b44985c12b01b66d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=556, earliestPutTs=1680062194327 2023-03-29 03:56:35,904 DEBUG [Thread-4303] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:56:35,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:56:35,907 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:56:35,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:56:35,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:56:35,916 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:56:35,916 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:35,916 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:56:35,916 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:56:35,916 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:35,917 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:35,918 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] throttle.PressureAwareThroughputController(145): 94b16ca160750195aa137348cdb80ba9#B#compaction#688 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:56:35,919 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:35,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:35,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:35,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:35,919 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:35,920 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:35,920 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:35,920 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:56:35,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:56:35,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:35,920 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:56:35,928 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:56:35,929 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:56:35,929 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:56:35,929 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:35,929 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:56:35,929 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:56:35,929 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:56:35,929 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:35,929 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. started... 2023-03-29 03:56:35,929 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 94b16ca160750195aa137348cdb80ba9 3/3 column families, dataSize=147.60 KB heapSize=414.61 KB 2023-03-29 03:56:35,930 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94b16ca160750195aa137348cdb80ba9, store=A 2023-03-29 03:56:35,930 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-29 03:56:35,930 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94b16ca160750195aa137348cdb80ba9, store=B 2023-03-29 03:56:35,930 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-29 03:56:35,930 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94b16ca160750195aa137348cdb80ba9, store=C 2023-03-29 03:56:35,930 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-29 03:56:35,933 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] throttle.PressureAwareThroughputController(145): 94b16ca160750195aa137348cdb80ba9#A#compaction#689 average throughput is 1.41 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:56:35,944 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/B/72766e198d6f4e288875beaaa949c941 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/72766e198d6f4e288875beaaa949c941 2023-03-29 03:56:35,949 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 94b16ca160750195aa137348cdb80ba9/B of 94b16ca160750195aa137348cdb80ba9 into 72766e198d6f4e288875beaaa949c941(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:56:35,949 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 94b16ca160750195aa137348cdb80ba9: 2023-03-29 03:56:35,949 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9., storeName=94b16ca160750195aa137348cdb80ba9/B, priority=12, startTime=1680062195899; duration=0sec 2023-03-29 03:56:35,949 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:56:35,949 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-29 03:56:35,950 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49732 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-29 03:56:35,950 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1912): 94b16ca160750195aa137348cdb80ba9/C is initiating minor compaction (all files) 2023-03-29 03:56:35,950 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 94b16ca160750195aa137348cdb80ba9/C in TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:35,950 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/93c93226f9fb473097488adce0ec58a4, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/93ad53cb25844d3ea023735a413b020a, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/a4671f180dc04808af5c7636ad4ea19c, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/713000b021b24cd4b0c15de1446c9c1a] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp, totalSize=48.6 K 2023-03-29 03:56:35,951 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 93c93226f9fb473097488adce0ec58a4, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=498, earliestPutTs=1680062192449 2023-03-29 03:56:35,951 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 93ad53cb25844d3ea023735a413b020a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=517, earliestPutTs=1680062192572 2023-03-29 03:56:35,951 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting a4671f180dc04808af5c7636ad4ea19c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=533, earliestPutTs=1680062193194 2023-03-29 03:56:35,952 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 713000b021b24cd4b0c15de1446c9c1a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=556, earliestPutTs=1680062194327 2023-03-29 03:56:35,977 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230329ed1c52eb594640c4a98637156b07857d to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230329ed1c52eb594640c4a98637156b07857d 2023-03-29 03:56:35,982 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] throttle.PressureAwareThroughputController(145): 94b16ca160750195aa137348cdb80ba9#C#compaction#691 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:56:36,414 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=610, memsize=49.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/A/d4f5c089004942d39f09ef4691b549af 2023-03-29 03:56:36,424 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=610 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/B/086af0428d4c41dca7ae6ae02ed8e0cc 2023-03-29 03:56:36,426 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/C/4c13dbbe82664e82b2b413594a35af19 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/4c13dbbe82664e82b2b413594a35af19 2023-03-29 03:56:36,431 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/A/1334983c9f60479e814150fcbd8422fc as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/1334983c9f60479e814150fcbd8422fc 2023-03-29 03:56:36,436 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=610 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/C/995bdb7bc9b84e8e97e59cc671d1b755 2023-03-29 03:56:36,436 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 94b16ca160750195aa137348cdb80ba9/C of 94b16ca160750195aa137348cdb80ba9 into 4c13dbbe82664e82b2b413594a35af19(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:56:36,436 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 94b16ca160750195aa137348cdb80ba9/A of 94b16ca160750195aa137348cdb80ba9 into 1334983c9f60479e814150fcbd8422fc(size=26.6 K), total size for store is 26.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:56:36,436 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 94b16ca160750195aa137348cdb80ba9: 2023-03-29 03:56:36,437 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 94b16ca160750195aa137348cdb80ba9: 2023-03-29 03:56:36,437 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9., storeName=94b16ca160750195aa137348cdb80ba9/C, priority=12, startTime=1680062195900; duration=0sec 2023-03-29 03:56:36,437 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9., storeName=94b16ca160750195aa137348cdb80ba9/A, priority=12, startTime=1680062195898; duration=0sec 2023-03-29 03:56:36,437 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:56:36,437 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:56:36,440 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/A/d4f5c089004942d39f09ef4691b549af as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/d4f5c089004942d39f09ef4691b549af 2023-03-29 03:56:36,443 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/d4f5c089004942d39f09ef4691b549af, entries=150, sequenceid=610, filesize=25.4 K 2023-03-29 03:56:36,444 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/B/086af0428d4c41dca7ae6ae02ed8e0cc as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/086af0428d4c41dca7ae6ae02ed8e0cc 2023-03-29 03:56:36,446 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/086af0428d4c41dca7ae6ae02ed8e0cc, entries=150, sequenceid=610, filesize=11.9 K 2023-03-29 03:56:36,447 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/C/995bdb7bc9b84e8e97e59cc671d1b755 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/995bdb7bc9b84e8e97e59cc671d1b755 2023-03-29 03:56:36,450 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/995bdb7bc9b84e8e97e59cc671d1b755, entries=150, sequenceid=610, filesize=11.9 K 2023-03-29 03:56:36,450 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~414.56 KB/424512, currentSize=0 B/0 for 94b16ca160750195aa137348cdb80ba9 in 521ms, sequenceid=610, compaction requested=false 2023-03-29 03:56:36,450 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 94b16ca160750195aa137348cdb80ba9: 2023-03-29 03:56:36,450 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:36,450 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:56:36,451 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:56:36,451 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:36,451 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:56:36,451 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:56:36,452 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:36,452 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:36,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:36,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:36,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:36,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:36,453 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:36,453 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:56:36,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:36,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:36,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:36,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:36,454 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:56:36,454 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4e10515c[Count = 0] remaining members to acquire global barrier 2023-03-29 03:56:36,454 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:56:36,454 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:36,455 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:36,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:36,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:36,456 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:56:36,456 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:56:36,456 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:36,456 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:56:36,456 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:56:36,458 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:56:36,458 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:36,459 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:56:36,459 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:36,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:36,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:36,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:36,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:36,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:36,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:36,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:36,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:36,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:36,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:36,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:56:36,461 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:56:36,461 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:56:36,461 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:56:36,461 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:56:36,461 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:56:36,461 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:56:36,465 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:36,465 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:36,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:36,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:36,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:36,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:36,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:36,465 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:36,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:36,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:36,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:36,466 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:36,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:36,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:36,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:36,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:36,467 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:36,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:36,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:36,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:36,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:36,469 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:36,469 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:36,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:36,469 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:36,469 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:36,469 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:36,469 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:56:36,469 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:56:36,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:36,469 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:36,470 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:56:36,470 DEBUG [Thread-4303] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:56:36,470 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:36,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:36,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:36,470 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:36,470 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:36,470 DEBUG [Thread-4303] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:56:36,470 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:36,570 DEBUG [Thread-4303] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:56:36,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:56:36,572 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:56:36,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:56:36,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:56:36,574 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:56:36,574 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:36,574 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:56:36,574 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:56:36,574 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:36,574 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:36,575 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:36,576 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:36,576 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:56:36,576 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:36,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:36,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:36,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:36,576 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:36,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:56:36,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:36,577 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:56:36,577 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:56:36,577 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:56:36,577 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:56:36,577 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:36,577 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:56:36,577 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:56:36,578 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:56:36,578 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:36,578 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. started... 2023-03-29 03:56:36,578 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 94b16ca160750195aa137348cdb80ba9: 2023-03-29 03:56:36,578 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:36,578 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:56:36,578 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:56:36,578 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:36,578 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:56:36,578 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:56:36,580 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:36,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:36,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:36,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:36,580 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:36,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:36,580 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:36,580 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:56:36,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:36,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:36,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:36,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:36,582 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:56:36,582 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@68de378b[Count = 0] remaining members to acquire global barrier 2023-03-29 03:56:36,582 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:56:36,582 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:36,584 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:36,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:36,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:36,584 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:56:36,584 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:56:36,584 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:56:36,584 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:36,584 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:56:36,585 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:56:36,585 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:36,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:36,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:36,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:36,585 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:36,586 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:56:36,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:36,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:36,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:36,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:36,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:36,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:36,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:36,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:56:36,588 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:56:36,588 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:56:36,588 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:56:36,588 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:56:36,588 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:56:36,588 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:56:36,590 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:36,590 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:36,590 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:36,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:36,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:36,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:36,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:36,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:36,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:36,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:36,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:36,591 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:36,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:36,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:36,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:36,592 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:36,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:36,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:36,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:36,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:36,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:36,594 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:36,594 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:36,594 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:36,594 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:56:36,594 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:56:36,594 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:36,594 DEBUG [Thread-4303] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:56:36,594 DEBUG [Thread-4303] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:56:36,594 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:36,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:36,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:36,594 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:36,595 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:56:36,594 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:36,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:36,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:36,595 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:36,595 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:36,595 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:36,694 DEBUG [Thread-4303] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:56:36,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:56:36,696 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:56:36,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:56:36,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:56:36,698 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:56:36,698 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:36,698 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:56:36,698 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:56:36,698 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:36,698 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:36,701 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:36,701 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:36,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:36,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:36,701 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:36,701 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:56:36,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:36,701 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:36,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:56:36,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:36,702 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:56:36,702 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:56:36,702 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:56:36,702 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:56:36,702 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:36,702 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:56:36,702 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:56:36,702 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:56:36,702 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:36,702 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. started... 2023-03-29 03:56:36,703 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 94b16ca160750195aa137348cdb80ba9: 2023-03-29 03:56:36,703 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:36,703 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:56:36,703 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:56:36,703 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:36,703 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:56:36,703 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:56:36,704 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:36,704 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:36,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:36,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:36,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:36,704 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:36,705 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:56:36,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:36,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:36,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:36,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:36,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:36,706 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:56:36,706 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@568cc144[Count = 0] remaining members to acquire global barrier 2023-03-29 03:56:36,706 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:56:36,706 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:36,707 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:36,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:36,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:36,707 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:56:36,707 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:56:36,707 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:56:36,707 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:36,707 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:56:36,709 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:56:36,709 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:36,709 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:36,709 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:56:36,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:36,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:36,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:36,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:36,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:36,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:36,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:36,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:36,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:36,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:36,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:56:36,711 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:56:36,711 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:56:36,711 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:56:36,711 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:56:36,711 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:56:36,711 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:56:36,713 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:36,714 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:36,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:36,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:36,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:36,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:36,714 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:36,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:36,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:36,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:36,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:36,714 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:36,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:36,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:36,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:36,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:36,715 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:36,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:36,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:36,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:36,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:36,718 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:36,718 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:36,718 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:36,718 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:36,718 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:56:36,718 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:36,718 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:36,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:36,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:36,718 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:36,718 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:56:36,718 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:36,718 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:36,718 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:36,719 DEBUG [Thread-4303] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:56:36,719 DEBUG [Thread-4303] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:56:36,719 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:56:36,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:36,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:36,819 DEBUG [Thread-4303] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:56:36,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:56:36,821 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:56:36,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:56:36,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:56:36,822 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:56:36,822 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:36,822 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:56:36,822 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:56:36,822 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:36,822 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:36,824 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:36,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:36,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:36,824 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:36,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:36,824 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:36,824 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:56:36,824 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:36,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:56:36,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:36,825 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:56:36,825 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:56:36,825 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:56:36,825 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:56:36,825 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:36,826 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:56:36,826 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:56:36,826 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:56:36,826 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:36,826 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. started... 2023-03-29 03:56:36,826 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 94b16ca160750195aa137348cdb80ba9: 2023-03-29 03:56:36,826 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:36,826 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:56:36,826 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:56:36,826 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:36,826 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:56:36,826 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:56:36,828 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:36,828 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:36,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:36,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:36,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:36,828 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:36,828 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:56:36,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:36,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:36,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:36,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:36,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:36,830 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:56:36,830 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f03fbbc[Count = 0] remaining members to acquire global barrier 2023-03-29 03:56:36,830 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:56:36,830 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:36,832 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:36,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:36,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:36,832 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:36,832 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:56:36,832 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:56:36,832 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:56:36,832 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:56:36,834 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:56:36,834 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:36,834 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:56:36,834 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:36,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:36,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:36,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:36,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:36,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:36,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:36,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:36,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:36,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:36,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:36,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:56:36,836 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:56:36,836 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:56:36,836 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:56:36,836 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:56:36,836 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:56:36,836 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:56:36,838 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:36,838 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:36,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:36,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:36,838 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:36,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:36,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:36,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:36,838 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:36,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:36,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:36,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:36,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:36,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:36,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:36,840 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:36,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:36,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:36,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:36,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:36,845 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:36,845 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:36,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:36,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:36,845 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:36,845 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:36,845 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:36,845 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:36,845 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:36,845 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:36,845 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:36,845 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:56:36,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:36,845 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:56:36,845 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:36,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:36,845 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:56:36,846 DEBUG [Thread-4303] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:56:36,846 DEBUG [Thread-4303] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:56:36,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on 94b16ca160750195aa137348cdb80ba9 2023-03-29 03:56:36,875 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 94b16ca160750195aa137348cdb80ba9 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-29 03:56:36,876 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94b16ca160750195aa137348cdb80ba9, store=A 2023-03-29 03:56:36,876 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:56:36,876 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94b16ca160750195aa137348cdb80ba9, store=B 2023-03-29 03:56:36,876 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:56:36,876 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94b16ca160750195aa137348cdb80ba9, store=C 2023-03-29 03:56:36,876 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:56:36,922 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:36,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37674 deadline: 1680062256920, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:36,923 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:36,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37666 deadline: 1680062256921, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:36,923 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680062196914, max timestamp=1680062196921Num uniques 859; 2023-03-29 03:56:36,923 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-29 03:56:36,923 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680062196914, max timestamp=1680062196921Num uniques 859; 2023-03-29 03:56:36,923 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-29 03:56:36,924 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680062196914, max timestamp=1680062196921Num uniques 859; 2023-03-29 03:56:36,924 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-29 03:56:36,946 DEBUG [Thread-4303] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:56:36,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:56:36,966 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:56:36,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:56:36,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:56:36,968 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:56:36,968 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:36,968 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:56:36,968 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:56:36,968 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:36,968 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:36,969 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:36,969 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:36,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:36,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:36,970 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:36,970 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:56:36,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:36,970 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:36,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:56:36,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:36,970 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:56:36,971 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:56:36,971 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:56:36,971 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:36,971 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:56:36,971 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:56:36,971 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:56:36,971 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:36,971 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. started... 2023-03-29 03:56:36,971 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. as already flushing 2023-03-29 03:56:36,971 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:36,971 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:56:36,971 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:56:36,971 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:36,971 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:56:36,971 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:56:36,972 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:36,973 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:36,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:36,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:36,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:36,973 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:36,973 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:56:36,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:36,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:36,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:36,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:36,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:36,974 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:56:36,974 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@33d5dfb1[Count = 0] remaining members to acquire global barrier 2023-03-29 03:56:36,974 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:56:36,975 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:36,976 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:36,976 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:36,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:36,976 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:56:36,976 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:56:36,976 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:56:36,976 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:36,976 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:56:36,978 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:56:36,978 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:36,978 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:56:36,978 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:36,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:36,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:36,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:36,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:36,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:36,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:36,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:36,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:36,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:36,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:36,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:56:36,981 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:56:36,981 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:56:36,981 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:56:36,981 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:56:36,981 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:56:36,981 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:56:36,982 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:36,982 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:36,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:36,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:36,982 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:36,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:36,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:36,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:36,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:36,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:36,983 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:36,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:36,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:36,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:36,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:36,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:36,984 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:36,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:36,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:36,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:36,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:36,987 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:36,987 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:36,987 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:56:36,987 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:56:36,987 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:36,987 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:36,987 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:36,987 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:36,987 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:36,987 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:36,987 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:36,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:36,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:36,987 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:36,987 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:56:36,987 DEBUG [Thread-4303] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:56:36,987 DEBUG [Thread-4303] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:56:36,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:36,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:36,993 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:36,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 257 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37712 deadline: 1680062256990, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:37,027 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:37,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37674 deadline: 1680062257024, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:37,027 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:37,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37666 deadline: 1680062257024, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:37,087 DEBUG [Thread-4303] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:56:37,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:56:37,089 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:56:37,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:56:37,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:56:37,091 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:56:37,091 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:37,091 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:56:37,091 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:56:37,091 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:37,091 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:37,092 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:37,092 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:37,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:37,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:37,093 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:37,093 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:56:37,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:37,093 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:37,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:56:37,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:37,094 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:56:37,094 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:56:37,094 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:56:37,094 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:56:37,094 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:37,094 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:56:37,094 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:56:37,094 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:56:37,094 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:37,095 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. started... 2023-03-29 03:56:37,095 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. as already flushing 2023-03-29 03:56:37,095 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:37,095 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:56:37,095 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:56:37,095 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:37,095 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:56:37,095 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:56:37,096 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:37,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 259 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37712 deadline: 1680062257094, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:37,097 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:37,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:37,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:37,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:37,097 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:37,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:37,097 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:37,097 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:56:37,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:37,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:37,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:37,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:37,099 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:56:37,099 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2aaed42f[Count = 0] remaining members to acquire global barrier 2023-03-29 03:56:37,099 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:56:37,099 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:37,100 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:37,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:37,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:37,100 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:56:37,100 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:56:37,100 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:56:37,100 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:37,100 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:56:37,101 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:37,101 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:56:37,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:37,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:37,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:37,101 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:37,101 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:56:37,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:37,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:37,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:37,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:37,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:37,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:37,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:37,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:56:37,103 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:56:37,103 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:56:37,103 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:56:37,104 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:56:37,104 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:56:37,104 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:56:37,105 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:37,105 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:37,105 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:37,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:37,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:37,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:37,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:37,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:37,106 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:37,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:37,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:37,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:37,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:37,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:37,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:37,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:37,107 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:37,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:37,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:37,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:37,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:37,110 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:37,110 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:37,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:37,110 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:56:37,110 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:56:37,110 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:37,110 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:37,110 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:37,110 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:37,110 DEBUG [Thread-4303] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:56:37,110 DEBUG [Thread-4303] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:56:37,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:37,110 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:37,110 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:37,111 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:37,111 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:37,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:37,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:37,111 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:56:37,211 DEBUG [Thread-4303] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:56:37,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:56:37,213 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:56:37,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:56:37,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:56:37,214 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:56:37,214 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:37,214 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:56:37,214 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:56:37,215 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:37,215 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:37,216 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:37,216 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:37,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:37,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:37,216 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:37,216 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:56:37,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:37,217 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:37,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:56:37,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:37,217 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:56:37,217 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:56:37,217 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:56:37,217 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:56:37,217 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:37,217 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:56:37,218 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:56:37,218 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:56:37,218 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:37,218 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. started... 2023-03-29 03:56:37,218 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. as already flushing 2023-03-29 03:56:37,218 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:37,218 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:56:37,218 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:56:37,218 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:37,218 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:56:37,218 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:56:37,220 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:37,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:37,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:37,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:37,220 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:37,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:37,221 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:37,221 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:56:37,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:37,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:37,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:37,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:37,222 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:56:37,222 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1a18b6df[Count = 0] remaining members to acquire global barrier 2023-03-29 03:56:37,222 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:56:37,222 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:37,223 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:37,223 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:37,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:37,223 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:56:37,223 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:56:37,223 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:56:37,224 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:37,224 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:56:37,226 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:56:37,226 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:37,226 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:37,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:37,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:37,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:37,226 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:56:37,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:37,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:37,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:37,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:37,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:37,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:37,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:37,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:56:37,229 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:56:37,229 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:56:37,229 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:56:37,229 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:56:37,229 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:56:37,229 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:56:37,230 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:37,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37666 deadline: 1680062257228, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:37,230 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:37,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37674 deadline: 1680062257229, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:37,231 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:37,231 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:37,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:37,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:37,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:37,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:37,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:37,231 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:37,232 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:37,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:37,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:37,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:37,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:37,232 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:37,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:37,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:37,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:37,234 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:37,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:37,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:37,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:37,236 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:37,237 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:37,237 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:37,236 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:37,237 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:56:37,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:37,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:37,237 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:56:37,237 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:56:37,237 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:37,237 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:37,237 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:37,237 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:37,237 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:37,237 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:37,237 DEBUG [Thread-4303] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:56:37,237 DEBUG [Thread-4303] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:56:37,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:37,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:37,293 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032960c727d994e14b36adbb57db96a8cd9d to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032960c727d994e14b36adbb57db96a8cd9d 2023-03-29 03:56:37,297 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=627, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/A/5fe313a6ed2b4b5a96df42c4312fbbdb 2023-03-29 03:56:37,300 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:37,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 261 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37712 deadline: 1680062257298, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:37,337 DEBUG [Thread-4303] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:56:37,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:56:37,339 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:56:37,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:56:37,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:56:37,340 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:56:37,340 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:37,340 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:56:37,341 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:56:37,341 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:37,341 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:37,348 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:37,348 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:37,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:37,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:37,349 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:37,349 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:56:37,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:37,349 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:37,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:56:37,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:37,349 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:56:37,350 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:56:37,350 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:56:37,350 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:56:37,350 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:37,350 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:56:37,350 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:56:37,350 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:56:37,350 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:37,350 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. started... 2023-03-29 03:56:37,350 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. as already flushing 2023-03-29 03:56:37,350 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:37,350 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:56:37,350 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:56:37,350 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:37,350 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:56:37,350 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:56:37,352 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:37,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:37,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:37,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:37,352 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:37,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:37,352 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:37,352 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:56:37,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:37,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:37,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:37,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:37,353 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:56:37,353 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7aa6703b[Count = 0] remaining members to acquire global barrier 2023-03-29 03:56:37,353 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:56:37,353 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:37,355 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:37,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:37,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:37,355 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:56:37,355 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:56:37,355 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:37,355 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:56:37,355 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:56:37,356 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:37,356 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:56:37,356 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:37,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:37,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:37,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:37,356 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:56:37,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:37,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:37,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:37,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:37,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:37,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:37,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:37,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:56:37,359 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:56:37,359 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:56:37,359 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:56:37,359 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:56:37,359 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:56:37,359 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:56:37,367 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:37,367 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:37,367 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:37,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:37,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:37,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:37,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:37,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:37,367 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:37,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:37,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:37,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:37,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:37,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:37,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:37,368 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:37,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:37,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:37,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:37,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:37,371 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:37,371 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:37,371 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:37,371 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:37,371 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:37,371 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:56:37,371 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:56:37,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:37,372 DEBUG [Thread-4303] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:56:37,372 DEBUG [Thread-4303] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:56:37,372 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:56:37,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:37,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:37,371 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:37,372 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:37,372 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:37,372 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:37,372 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:37,372 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:37,472 DEBUG [Thread-4303] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:56:37,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:56:37,474 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:56:37,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:56:37,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:56:37,476 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:56:37,476 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:37,476 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:56:37,476 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:56:37,477 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:37,477 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:37,478 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:37,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:37,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:37,478 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:37,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:37,478 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:37,478 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:56:37,479 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:37,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:56:37,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:37,479 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:56:37,479 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:56:37,479 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:56:37,479 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:56:37,480 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:37,480 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:56:37,480 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:56:37,480 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:56:37,480 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:37,480 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. started... 2023-03-29 03:56:37,480 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. as already flushing 2023-03-29 03:56:37,480 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:37,480 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:56:37,480 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:56:37,480 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:37,480 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:56:37,480 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:56:37,482 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:37,483 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:37,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:37,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:37,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:37,483 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:37,483 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:56:37,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:37,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:37,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:37,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:37,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:37,485 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:56:37,485 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@67130089[Count = 0] remaining members to acquire global barrier 2023-03-29 03:56:37,485 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:56:37,485 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:37,490 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:37,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:37,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:37,491 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:56:37,491 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:56:37,491 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:56:37,491 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:37,491 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:56:37,493 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:37,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:37,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:37,493 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:56:37,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:37,493 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:37,493 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:56:37,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:37,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:37,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:37,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:37,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:37,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:37,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:37,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:56:37,498 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:56:37,498 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:56:37,498 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:56:37,498 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:56:37,498 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:56:37,498 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:56:37,499 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:37,499 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:37,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:37,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:37,499 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:37,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:37,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:37,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:37,500 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:37,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:37,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:37,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:37,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:37,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:37,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:37,501 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:37,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:37,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:37,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:37,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:37,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:37,503 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:37,503 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:37,503 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:37,503 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:37,503 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:37,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:37,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:37,503 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:56:37,504 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:56:37,504 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:56:37,503 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:37,504 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:37,504 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:37,504 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:37,504 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:37,504 DEBUG [Thread-4303] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:56:37,504 DEBUG [Thread-4303] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:56:37,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:37,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:37,534 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:37,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37666 deadline: 1680062257531, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:37,534 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:37,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 246 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37674 deadline: 1680062257532, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:37,602 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:37,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 263 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37712 deadline: 1680062257602, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:37,604 DEBUG [Thread-4303] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:56:37,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:56:37,606 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:56:37,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:56:37,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:56:37,607 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:56:37,608 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:37,608 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:56:37,608 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:56:37,608 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:37,608 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:37,610 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:37,610 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:37,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:37,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:37,611 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:37,611 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:56:37,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:37,611 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:37,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:56:37,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:37,611 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:56:37,612 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:56:37,612 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:56:37,612 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:56:37,612 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:37,612 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:56:37,612 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:56:37,612 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:56:37,612 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:37,612 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. started... 2023-03-29 03:56:37,612 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. as already flushing 2023-03-29 03:56:37,612 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:37,613 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:56:37,613 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:56:37,613 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:37,613 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:56:37,613 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:56:37,614 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:37,614 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:37,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:37,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:37,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:37,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:37,615 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:37,615 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:56:37,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:37,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:37,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:37,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:37,617 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:56:37,617 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3b40bfe8[Count = 0] remaining members to acquire global barrier 2023-03-29 03:56:37,617 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:56:37,617 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:37,618 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:37,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:37,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:37,618 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:56:37,618 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:56:37,618 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:56:37,618 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:37,619 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:56:37,620 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:56:37,620 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:37,620 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:56:37,620 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:37,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:37,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:37,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:37,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:37,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:37,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:37,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:37,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:37,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:37,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:37,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:56:37,623 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:56:37,624 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:56:37,624 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:56:37,624 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:56:37,624 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:56:37,624 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:56:37,630 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:37,630 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:37,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:37,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:37,630 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:37,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:37,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:37,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:37,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:37,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:37,631 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:37,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:37,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:37,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:37,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:37,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:37,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:37,632 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:37,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:37,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:37,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:37,635 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:37,635 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:37,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:37,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:37,636 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:56:37,636 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:56:37,636 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:37,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:37,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:37,636 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:37,636 DEBUG [Thread-4303] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:56:37,636 DEBUG [Thread-4303] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:56:37,636 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:37,636 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:37,636 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:37,636 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:37,636 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:37,636 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:37,707 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=627 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/B/1240308bce954b3fae49e93edf24cbc3 2023-03-29 03:56:37,715 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=627 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/C/00490b2eb48a4d498cf6224cfd1ebb49 2023-03-29 03:56:37,719 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/A/5fe313a6ed2b4b5a96df42c4312fbbdb as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/5fe313a6ed2b4b5a96df42c4312fbbdb 2023-03-29 03:56:37,722 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/5fe313a6ed2b4b5a96df42c4312fbbdb, entries=350, sequenceid=627, filesize=52.9 K 2023-03-29 03:56:37,723 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/B/1240308bce954b3fae49e93edf24cbc3 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/1240308bce954b3fae49e93edf24cbc3 2023-03-29 03:56:37,726 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/1240308bce954b3fae49e93edf24cbc3, entries=150, sequenceid=627, filesize=11.9 K 2023-03-29 03:56:37,726 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/C/00490b2eb48a4d498cf6224cfd1ebb49 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/00490b2eb48a4d498cf6224cfd1ebb49 2023-03-29 03:56:37,729 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/00490b2eb48a4d498cf6224cfd1ebb49, entries=150, sequenceid=627, filesize=11.9 K 2023-03-29 03:56:37,730 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 94b16ca160750195aa137348cdb80ba9 in 855ms, sequenceid=627, compaction requested=true 2023-03-29 03:56:37,730 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 94b16ca160750195aa137348cdb80ba9: 2023-03-29 03:56:37,730 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:56:37,730 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-29 03:56:37,730 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-29 03:56:37,730 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:56:37,730 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:56:37,731 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 107464 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-29 03:56:37,731 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37682 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-29 03:56:37,731 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1912): 94b16ca160750195aa137348cdb80ba9/A is initiating minor compaction (all files) 2023-03-29 03:56:37,731 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1912): 94b16ca160750195aa137348cdb80ba9/B is initiating minor compaction (all files) 2023-03-29 03:56:37,731 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 94b16ca160750195aa137348cdb80ba9/A in TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:37,731 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 94b16ca160750195aa137348cdb80ba9/B in TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:37,731 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/1334983c9f60479e814150fcbd8422fc, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/d4f5c089004942d39f09ef4691b549af, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/5fe313a6ed2b4b5a96df42c4312fbbdb] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp, totalSize=104.9 K 2023-03-29 03:56:37,732 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/72766e198d6f4e288875beaaa949c941, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/086af0428d4c41dca7ae6ae02ed8e0cc, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/1240308bce954b3fae49e93edf24cbc3] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp, totalSize=36.8 K 2023-03-29 03:56:37,732 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 1334983c9f60479e814150fcbd8422fc, keycount=150, bloomtype=ROW, size=26.6 K, encoding=NONE, compression=NONE, seqNum=556, earliestPutTs=1680062194327 2023-03-29 03:56:37,732 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 72766e198d6f4e288875beaaa949c941, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=556, earliestPutTs=1680062194327 2023-03-29 03:56:37,732 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting d4f5c089004942d39f09ef4691b549af, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=610, earliestPutTs=1680062194718 2023-03-29 03:56:37,732 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 086af0428d4c41dca7ae6ae02ed8e0cc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=610, earliestPutTs=1680062194718 2023-03-29 03:56:37,733 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 5fe313a6ed2b4b5a96df42c4312fbbdb, keycount=350, bloomtype=ROW, size=52.9 K, encoding=NONE, compression=NONE, seqNum=627, earliestPutTs=1680062196861 2023-03-29 03:56:37,733 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 1240308bce954b3fae49e93edf24cbc3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=627, earliestPutTs=1680062196871 2023-03-29 03:56:37,736 DEBUG [Thread-4303] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:56:37,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:56:37,738 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] throttle.PressureAwareThroughputController(145): 94b16ca160750195aa137348cdb80ba9#B#compaction#698 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:56:37,740 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:56:37,741 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] throttle.PressureAwareThroughputController(145): 94b16ca160750195aa137348cdb80ba9#A#compaction#697 average throughput is 4.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:56:37,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:56:37,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:56:37,748 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:56:37,748 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:37,748 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:56:37,748 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:56:37,748 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:37,748 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:37,749 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:37,749 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:37,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:37,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:37,750 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:37,750 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:56:37,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:37,750 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:37,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:56:37,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:37,750 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:56:37,757 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:56:37,757 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:56:37,757 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:56:37,757 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:37,757 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:56:37,758 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:56:37,758 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:56:37,758 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:37,758 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. started... 2023-03-29 03:56:37,758 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 94b16ca160750195aa137348cdb80ba9 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-03-29 03:56:37,758 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94b16ca160750195aa137348cdb80ba9, store=A 2023-03-29 03:56:37,759 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-29 03:56:37,759 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94b16ca160750195aa137348cdb80ba9, store=B 2023-03-29 03:56:37,759 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-29 03:56:37,759 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94b16ca160750195aa137348cdb80ba9, store=C 2023-03-29 03:56:37,759 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-29 03:56:37,765 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/B/e7765708c0d24c55bb0cc39c85057bd4 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/e7765708c0d24c55bb0cc39c85057bd4 2023-03-29 03:56:37,773 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 94b16ca160750195aa137348cdb80ba9/B of 94b16ca160750195aa137348cdb80ba9 into e7765708c0d24c55bb0cc39c85057bd4(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:56:37,774 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 94b16ca160750195aa137348cdb80ba9: 2023-03-29 03:56:37,774 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9., storeName=94b16ca160750195aa137348cdb80ba9/B, priority=13, startTime=1680062197730; duration=0sec 2023-03-29 03:56:37,774 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:56:37,774 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-29 03:56:37,774 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37682 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-29 03:56:37,775 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1912): 94b16ca160750195aa137348cdb80ba9/C is initiating minor compaction (all files) 2023-03-29 03:56:37,775 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 94b16ca160750195aa137348cdb80ba9/C in TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:37,775 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/4c13dbbe82664e82b2b413594a35af19, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/995bdb7bc9b84e8e97e59cc671d1b755, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/00490b2eb48a4d498cf6224cfd1ebb49] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp, totalSize=36.8 K 2023-03-29 03:56:37,775 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 4c13dbbe82664e82b2b413594a35af19, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=556, earliestPutTs=1680062194327 2023-03-29 03:56:37,775 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 995bdb7bc9b84e8e97e59cc671d1b755, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=610, earliestPutTs=1680062194718 2023-03-29 03:56:37,776 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 00490b2eb48a4d498cf6224cfd1ebb49, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=627, earliestPutTs=1680062196871 2023-03-29 03:56:37,781 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] throttle.PressureAwareThroughputController(145): 94b16ca160750195aa137348cdb80ba9#C#compaction#700 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:56:37,788 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230329c3323a0b271f4d24825c77a272145ee9 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230329c3323a0b271f4d24825c77a272145ee9 2023-03-29 03:56:37,791 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/C/05992b62470540778f3858dba335af3b as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/05992b62470540778f3858dba335af3b 2023-03-29 03:56:37,795 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 94b16ca160750195aa137348cdb80ba9/C of 94b16ca160750195aa137348cdb80ba9 into 05992b62470540778f3858dba335af3b(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:56:37,795 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 94b16ca160750195aa137348cdb80ba9: 2023-03-29 03:56:37,795 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9., storeName=94b16ca160750195aa137348cdb80ba9/C, priority=13, startTime=1680062197730; duration=0sec 2023-03-29 03:56:37,795 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:56:37,801 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=663, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/A/3d0083d5c2fb42a4b3440162bcb97267 2023-03-29 03:56:38,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on 94b16ca160750195aa137348cdb80ba9 2023-03-29 03:56:38,064 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. as already flushing 2023-03-29 03:56:38,098 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:38,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37666 deadline: 1680062258095, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:38,099 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:38,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37674 deadline: 1680062258095, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:38,107 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:38,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 265 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37712 deadline: 1680062258106, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:38,188 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/A/54f9e76b5c004f91ae3defa3cae79fff as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/54f9e76b5c004f91ae3defa3cae79fff 2023-03-29 03:56:38,194 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 94b16ca160750195aa137348cdb80ba9/A of 94b16ca160750195aa137348cdb80ba9 into 54f9e76b5c004f91ae3defa3cae79fff(size=26.7 K), total size for store is 26.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:56:38,194 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 94b16ca160750195aa137348cdb80ba9: 2023-03-29 03:56:38,194 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9., storeName=94b16ca160750195aa137348cdb80ba9/A, priority=13, startTime=1680062197730; duration=0sec 2023-03-29 03:56:38,194 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:56:38,201 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:38,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37666 deadline: 1680062258200, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:38,203 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:38,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37674 deadline: 1680062258200, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:38,239 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=663 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/B/4c49bd9cc7b947c5bbd96a75e8196a72 2023-03-29 03:56:38,407 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:38,407 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:38,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37666 deadline: 1680062258404, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:38,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 257 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37674 deadline: 1680062258404, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:38,541 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:38,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37652 deadline: 1680062258539, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:38,541 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:38,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37670 deadline: 1680062258539, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:38,543 DEBUG [Thread-4299] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8163 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9., hostname=jenkins-hbase4.apache.org,35599,1680062018534, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-29 03:56:38,544 DEBUG [Thread-4297] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8169 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9., hostname=jenkins-hbase4.apache.org,35599,1680062018534, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-29 03:56:38,648 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=663 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/C/c848fe1465024bf390f67d59d90b3548 2023-03-29 03:56:38,652 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/A/3d0083d5c2fb42a4b3440162bcb97267 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/3d0083d5c2fb42a4b3440162bcb97267 2023-03-29 03:56:38,655 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/3d0083d5c2fb42a4b3440162bcb97267, entries=150, sequenceid=663, filesize=25.4 K 2023-03-29 03:56:38,655 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/B/4c49bd9cc7b947c5bbd96a75e8196a72 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/4c49bd9cc7b947c5bbd96a75e8196a72 2023-03-29 03:56:38,659 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/4c49bd9cc7b947c5bbd96a75e8196a72, entries=150, sequenceid=663, filesize=11.9 K 2023-03-29 03:56:38,659 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/C/c848fe1465024bf390f67d59d90b3548 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/c848fe1465024bf390f67d59d90b3548 2023-03-29 03:56:38,662 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/c848fe1465024bf390f67d59d90b3548, entries=150, sequenceid=663, filesize=11.9 K 2023-03-29 03:56:38,662 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=60.38 KB/61830 for 94b16ca160750195aa137348cdb80ba9 in 904ms, sequenceid=663, compaction requested=false 2023-03-29 03:56:38,662 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 94b16ca160750195aa137348cdb80ba9: 2023-03-29 03:56:38,662 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:38,662 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:56:38,663 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:56:38,663 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:38,663 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:56:38,663 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:56:38,667 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:38,667 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:38,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:38,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:38,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:38,668 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:38,668 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:56:38,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:38,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:38,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:38,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:38,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:38,669 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:56:38,670 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@64314811[Count = 0] remaining members to acquire global barrier 2023-03-29 03:56:38,670 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:56:38,670 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:38,671 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:38,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:38,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:38,671 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:56:38,671 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:56:38,671 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:56:38,671 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:38,671 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:56:38,673 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:38,673 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:56:38,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:38,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:38,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:38,673 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:38,673 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:56:38,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:38,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:38,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:38,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:38,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:38,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:38,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:38,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:56:38,675 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:56:38,675 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:56:38,675 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:56:38,675 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:56:38,675 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:56:38,675 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:56:38,677 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:38,677 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:38,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:38,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:38,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:38,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:38,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:38,677 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:38,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:38,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:38,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:38,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:38,678 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:38,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:38,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:38,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:38,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:38,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:38,679 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:38,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:38,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:38,682 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:38,682 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:38,682 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:38,682 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:38,682 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:38,682 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:38,682 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:38,688 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:38,688 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:38,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:38,688 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:38,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:38,688 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:56:38,688 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:56:38,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:38,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:38,688 DEBUG [Thread-4303] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:56:38,688 DEBUG [Thread-4303] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:56:38,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on 94b16ca160750195aa137348cdb80ba9 2023-03-29 03:56:38,710 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 94b16ca160750195aa137348cdb80ba9 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-29 03:56:38,710 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94b16ca160750195aa137348cdb80ba9, store=A 2023-03-29 03:56:38,710 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:56:38,710 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94b16ca160750195aa137348cdb80ba9, store=B 2023-03-29 03:56:38,710 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:56:38,710 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94b16ca160750195aa137348cdb80ba9, store=C 2023-03-29 03:56:38,710 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:56:38,721 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230329d5607cbb334b426bb95a7e98350b3ed8 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230329d5607cbb334b426bb95a7e98350b3ed8 2023-03-29 03:56:38,725 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=679, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/A/0d091bd580d34876845e40599a3ed49a 2023-03-29 03:56:38,734 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=679 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/B/acbc9e9359cb4536b40a0957087c8beb 2023-03-29 03:56:38,737 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:38,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 267 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37674 deadline: 1680062258733, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:38,741 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=679 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/C/9035f3877f4a401b82e2ab0cb928a270 2023-03-29 03:56:38,741 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:38,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 260 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37666 deadline: 1680062258737, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:38,744 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/A/0d091bd580d34876845e40599a3ed49a as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/0d091bd580d34876845e40599a3ed49a 2023-03-29 03:56:38,747 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/0d091bd580d34876845e40599a3ed49a, entries=200, sequenceid=679, filesize=32.3 K 2023-03-29 03:56:38,747 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/B/acbc9e9359cb4536b40a0957087c8beb as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/acbc9e9359cb4536b40a0957087c8beb 2023-03-29 03:56:38,750 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/acbc9e9359cb4536b40a0957087c8beb, entries=150, sequenceid=679, filesize=11.9 K 2023-03-29 03:56:38,751 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/C/9035f3877f4a401b82e2ab0cb928a270 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/9035f3877f4a401b82e2ab0cb928a270 2023-03-29 03:56:38,754 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/9035f3877f4a401b82e2ab0cb928a270, entries=150, sequenceid=679, filesize=11.9 K 2023-03-29 03:56:38,756 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 94b16ca160750195aa137348cdb80ba9 in 46ms, sequenceid=679, compaction requested=true 2023-03-29 03:56:38,756 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 94b16ca160750195aa137348cdb80ba9: 2023-03-29 03:56:38,756 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:56:38,756 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-29 03:56:38,756 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-29 03:56:38,756 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:56:38,756 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:56:38,757 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 86438 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-29 03:56:38,757 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1912): 94b16ca160750195aa137348cdb80ba9/A is initiating minor compaction (all files) 2023-03-29 03:56:38,757 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 94b16ca160750195aa137348cdb80ba9/A in TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:38,757 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/54f9e76b5c004f91ae3defa3cae79fff, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/3d0083d5c2fb42a4b3440162bcb97267, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/0d091bd580d34876845e40599a3ed49a] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp, totalSize=84.4 K 2023-03-29 03:56:38,758 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37784 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-29 03:56:38,758 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1912): 94b16ca160750195aa137348cdb80ba9/B is initiating minor compaction (all files) 2023-03-29 03:56:38,758 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 94b16ca160750195aa137348cdb80ba9/B in TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:38,758 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/e7765708c0d24c55bb0cc39c85057bd4, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/4c49bd9cc7b947c5bbd96a75e8196a72, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/acbc9e9359cb4536b40a0957087c8beb] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp, totalSize=36.9 K 2023-03-29 03:56:38,758 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 54f9e76b5c004f91ae3defa3cae79fff, keycount=150, bloomtype=ROW, size=26.7 K, encoding=NONE, compression=NONE, seqNum=627, earliestPutTs=1680062196871 2023-03-29 03:56:38,758 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting e7765708c0d24c55bb0cc39c85057bd4, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=627, earliestPutTs=1680062196871 2023-03-29 03:56:38,759 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 4c49bd9cc7b947c5bbd96a75e8196a72, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=663, earliestPutTs=1680062196982 2023-03-29 03:56:38,759 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 3d0083d5c2fb42a4b3440162bcb97267, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=663, earliestPutTs=1680062196982 2023-03-29 03:56:38,759 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting acbc9e9359cb4536b40a0957087c8beb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=679, earliestPutTs=1680062198040 2023-03-29 03:56:38,759 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 0d091bd580d34876845e40599a3ed49a, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=679, earliestPutTs=1680062198040 2023-03-29 03:56:38,768 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] throttle.PressureAwareThroughputController(145): 94b16ca160750195aa137348cdb80ba9#B#compaction#706 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:56:38,770 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] throttle.PressureAwareThroughputController(145): 94b16ca160750195aa137348cdb80ba9#A#compaction#707 average throughput is 6.57 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:56:38,788 DEBUG [Thread-4303] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:56:38,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:56:38,790 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:56:38,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:56:38,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:56:38,792 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:56:38,792 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:38,792 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:56:38,792 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:56:38,792 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:38,792 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:38,794 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:38,794 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:38,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:38,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:38,794 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:38,794 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:56:38,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:38,794 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:38,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:56:38,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:38,795 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:56:38,795 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:56:38,795 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:56:38,795 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:56:38,795 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:38,795 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:56:38,795 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:56:38,795 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:56:38,795 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:38,795 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. started... 2023-03-29 03:56:38,795 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 94b16ca160750195aa137348cdb80ba9 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-29 03:56:38,796 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94b16ca160750195aa137348cdb80ba9, store=A 2023-03-29 03:56:38,796 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:56:38,796 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94b16ca160750195aa137348cdb80ba9, store=B 2023-03-29 03:56:38,796 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:56:38,796 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94b16ca160750195aa137348cdb80ba9, store=C 2023-03-29 03:56:38,796 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:56:38,822 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230329e26150dc8740489f8769c6526c37f590 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230329e26150dc8740489f8769c6526c37f590 2023-03-29 03:56:38,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on 94b16ca160750195aa137348cdb80ba9 2023-03-29 03:56:38,840 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. as already flushing 2023-03-29 03:56:38,843 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=698, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/A/5ee3ad1521a14e81a1dae3a9d6f0701b 2023-03-29 03:56:38,853 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=698 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/B/5c8e4a38f4b448849d207223f33fb8a2 2023-03-29 03:56:38,861 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:38,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 266 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37666 deadline: 1680062258858, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:38,863 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:38,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] ipc.CallRunner(144): callId: 275 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37674 deadline: 1680062258860, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:38,865 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=698 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/C/2fae292ba5464616ad301a39e3bb3cb6 2023-03-29 03:56:38,869 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/A/5ee3ad1521a14e81a1dae3a9d6f0701b as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/5ee3ad1521a14e81a1dae3a9d6f0701b 2023-03-29 03:56:38,872 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/5ee3ad1521a14e81a1dae3a9d6f0701b, entries=150, sequenceid=698, filesize=25.4 K 2023-03-29 03:56:38,873 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/B/5c8e4a38f4b448849d207223f33fb8a2 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/5c8e4a38f4b448849d207223f33fb8a2 2023-03-29 03:56:38,876 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/5c8e4a38f4b448849d207223f33fb8a2, entries=150, sequenceid=698, filesize=11.9 K 2023-03-29 03:56:38,876 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/C/2fae292ba5464616ad301a39e3bb3cb6 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/2fae292ba5464616ad301a39e3bb3cb6 2023-03-29 03:56:38,880 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/2fae292ba5464616ad301a39e3bb3cb6, entries=150, sequenceid=698, filesize=11.9 K 2023-03-29 03:56:38,881 INFO [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 94b16ca160750195aa137348cdb80ba9 in 86ms, sequenceid=698, compaction requested=true 2023-03-29 03:56:38,881 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 94b16ca160750195aa137348cdb80ba9: 2023-03-29 03:56:38,881 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:38,881 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:56:38,881 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:56:38,881 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:38,881 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:56:38,881 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:56:38,883 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:38,883 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:38,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:38,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:38,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:38,883 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:38,883 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:56:38,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:38,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:38,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:38,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:38,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:38,885 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:56:38,885 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d5fcb04[Count = 0] remaining members to acquire global barrier 2023-03-29 03:56:38,885 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:56:38,885 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:38,886 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:38,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:38,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:38,886 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:56:38,886 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:56:38,886 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:56:38,886 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:38,887 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:56:38,889 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:56:38,889 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:38,889 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:38,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:38,889 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:56:38,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:38,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:38,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:38,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:38,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:38,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:38,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:38,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:38,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:38,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:56:38,891 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:56:38,891 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:56:38,891 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:56:38,891 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:56:38,891 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:56:38,891 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:56:38,893 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:38,893 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:38,893 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:38,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:38,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:38,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:38,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:38,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:38,893 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:38,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:38,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:38,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:38,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:38,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:38,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:38,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:38,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:38,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:38,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:38,896 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:38,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:38,898 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:38,898 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:38,898 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:38,898 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:38,899 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:38,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:38,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:38,899 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:38,899 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:38,899 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:38,899 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:56:38,899 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:56:38,899 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:38,899 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:38,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:38,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:38,899 DEBUG [Thread-4303] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:56:38,899 DEBUG [Thread-4303] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:56:38,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on 94b16ca160750195aa137348cdb80ba9 2023-03-29 03:56:38,963 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 94b16ca160750195aa137348cdb80ba9 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-03-29 03:56:38,963 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94b16ca160750195aa137348cdb80ba9, store=A 2023-03-29 03:56:38,963 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:56:38,963 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94b16ca160750195aa137348cdb80ba9, store=B 2023-03-29 03:56:38,963 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:56:38,963 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94b16ca160750195aa137348cdb80ba9, store=C 2023-03-29 03:56:38,963 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:56:38,974 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230329ebf03b4f56ea401584ef8432a8d06e1a to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230329ebf03b4f56ea401584ef8432a8d06e1a 2023-03-29 03:56:38,999 DEBUG [Thread-4303] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:56:39,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:56:39,001 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-29 03:56:39,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-29 03:56:39,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-29 03:56:39,002 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-29 03:56:39,003 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:39,003 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-29 03:56:39,003 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-29 03:56:39,003 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:39,003 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:39,004 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:39,004 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:39,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:39,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:39,004 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:39,004 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-29 03:56:39,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:39,005 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:39,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-29 03:56:39,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:39,005 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-29 03:56:39,005 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-29 03:56:39,005 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-29 03:56:39,005 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-29 03:56:39,005 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-29 03:56:39,005 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-29 03:56:39,006 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-29 03:56:39,006 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-29 03:56:39,006 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:39,006 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. started... 2023-03-29 03:56:39,006 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. as already flushing 2023-03-29 03:56:39,006 DEBUG [rs(jenkins-hbase4.apache.org,35599,1680062018534)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:39,006 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-29 03:56:39,006 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-29 03:56:39,006 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:39,006 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-29 03:56:39,006 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-29 03:56:39,006 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:39,006 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:39,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 277 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37666 deadline: 1680062259002, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:39,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 285 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37674 deadline: 1680062259002, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:39,008 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:39,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:39,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:39,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:39,008 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:39,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:39,009 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:39,009 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-29 03:56:39,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:39,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:39,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:39,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:39,010 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35599,1680062018534' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-29 03:56:39,010 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5237916e[Count = 0] remaining members to acquire global barrier 2023-03-29 03:56:39,010 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-29 03:56:39,010 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:39,011 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:39,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:39,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:39,011 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-29 03:56:39,012 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-29 03:56:39,012 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:39,012 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-29 03:56:39,012 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35599,1680062018534' in zk 2023-03-29 03:56:39,013 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-29 03:56:39,013 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:39,013 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:39,013 DEBUG [member: 'jenkins-hbase4.apache.org,35599,1680062018534' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-29 03:56:39,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:39,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:39,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:39,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:39,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:39,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:39,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:39,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:39,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:39,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:39,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35599,1680062018534': 2023-03-29 03:56:39,015 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35599,1680062018534' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-29 03:56:39,015 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-29 03:56:39,015 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-29 03:56:39,015 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-29 03:56:39,015 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-29 03:56:39,015 INFO [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-29 03:56:39,017 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:39,017 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:39,017 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:39,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:39,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:39,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:39,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-29 03:56:39,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-29 03:56:39,018 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:39,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:39,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:39,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-29 03:56:39,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:39,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:39,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-29 03:56:39,019 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:39,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:39,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:39,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-29 03:56:39,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-29 03:56:39,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:39,023 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:39,024 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:39,024 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-29 03:56:39,024 DEBUG [Thread-4308] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3dca23c0 to 127.0.0.1:52103 2023-03-29 03:56:39,024 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:39,024 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:39,024 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-29 03:56:39,024 DEBUG [Thread-4308] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-29 03:56:39,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-29 03:56:39,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-29 03:56:39,024 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-29 03:56:39,024 DEBUG [Thread-4312] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2a8bf279 to 127.0.0.1:52103 2023-03-29 03:56:39,024 DEBUG [Thread-4312] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-29 03:56:39,024 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-29 03:56:39,024 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-29 03:56:39,024 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-29 03:56:39,024 DEBUG [(jenkins-hbase4.apache.org,43505,1680062017306)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-29 03:56:39,024 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-29 03:56:39,025 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-29 03:56:39,026 DEBUG [Thread-4304] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0a470a3d to 127.0.0.1:52103 2023-03-29 03:56:39,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-29 03:56:39,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-29 03:56:39,026 DEBUG [Thread-4304] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-29 03:56:39,027 DEBUG [Thread-4303] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-29 03:56:39,027 DEBUG [Thread-4303] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-29 03:56:39,028 DEBUG [Thread-4310] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x62d52321 to 127.0.0.1:52103 2023-03-29 03:56:39,028 DEBUG [Thread-4306] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4bb72437 to 127.0.0.1:52103 2023-03-29 03:56:39,028 DEBUG [Thread-4310] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-29 03:56:39,028 DEBUG [Thread-4306] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-29 03:56:39,108 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:39,108 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:39,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35599] ipc.CallRunner(144): callId: 279 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37666 deadline: 1680062259108, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:39,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 287 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37674 deadline: 1680062259108, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:39,111 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:39,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 267 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37712 deadline: 1680062259111, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:39,127 DEBUG [Thread-4303] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-29 03:56:39,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-29 03:56:39,178 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/B/f856925c8d034f8eb1b3d335aa2fa17b as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/f856925c8d034f8eb1b3d335aa2fa17b 2023-03-29 03:56:39,181 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 94b16ca160750195aa137348cdb80ba9/B of 94b16ca160750195aa137348cdb80ba9 into f856925c8d034f8eb1b3d335aa2fa17b(size=13.2 K), total size for store is 25.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:56:39,181 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 94b16ca160750195aa137348cdb80ba9: 2023-03-29 03:56:39,181 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9., storeName=94b16ca160750195aa137348cdb80ba9/B, priority=13, startTime=1680062198756; duration=0sec 2023-03-29 03:56:39,181 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:56:39,181 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-29 03:56:39,182 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49970 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-29 03:56:39,182 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1912): 94b16ca160750195aa137348cdb80ba9/C is initiating minor compaction (all files) 2023-03-29 03:56:39,182 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 94b16ca160750195aa137348cdb80ba9/C in TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:39,182 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/05992b62470540778f3858dba335af3b, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/c848fe1465024bf390f67d59d90b3548, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/9035f3877f4a401b82e2ab0cb928a270, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/2fae292ba5464616ad301a39e3bb3cb6] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp, totalSize=48.8 K 2023-03-29 03:56:39,182 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 05992b62470540778f3858dba335af3b, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=627, earliestPutTs=1680062196871 2023-03-29 03:56:39,182 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting c848fe1465024bf390f67d59d90b3548, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=663, earliestPutTs=1680062196982 2023-03-29 03:56:39,182 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 9035f3877f4a401b82e2ab0cb928a270, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=679, earliestPutTs=1680062198040 2023-03-29 03:56:39,183 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 2fae292ba5464616ad301a39e3bb3cb6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=698, earliestPutTs=1680062198732 2023-03-29 03:56:39,187 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] throttle.PressureAwareThroughputController(145): 94b16ca160750195aa137348cdb80ba9#C#compaction#712 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:56:39,194 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/C/49f3abbe870b48dd8be4abb9c9c02cfc as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/49f3abbe870b48dd8be4abb9c9c02cfc 2023-03-29 03:56:39,195 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/A/6ec31d144bea47c18d0932199581adba as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/6ec31d144bea47c18d0932199581adba 2023-03-29 03:56:39,197 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 94b16ca160750195aa137348cdb80ba9/C of 94b16ca160750195aa137348cdb80ba9 into 49f3abbe870b48dd8be4abb9c9c02cfc(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:56:39,197 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 94b16ca160750195aa137348cdb80ba9: 2023-03-29 03:56:39,197 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9., storeName=94b16ca160750195aa137348cdb80ba9/C, priority=12, startTime=1680062198756; duration=0sec 2023-03-29 03:56:39,197 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:56:39,199 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 94b16ca160750195aa137348cdb80ba9/A of 94b16ca160750195aa137348cdb80ba9 into 6ec31d144bea47c18d0932199581adba(size=26.8 K), total size for store is 52.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:56:39,199 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 94b16ca160750195aa137348cdb80ba9: 2023-03-29 03:56:39,199 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9., storeName=94b16ca160750195aa137348cdb80ba9/A, priority=13, startTime=1680062198756; duration=0sec 2023-03-29 03:56:39,199 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:56:39,309 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:39,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 281 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37666 deadline: 1680062259309, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:39,310 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:39,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 289 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37674 deadline: 1680062259310, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:39,380 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=712, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/A/2bfd75892d3a4e31bc32027b382accd1 2023-03-29 03:56:39,611 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:39,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 283 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37666 deadline: 1680062259611, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:39,611 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:39,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 291 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37674 deadline: 1680062259611, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:39,788 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=712 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/B/e28748f201a84445b832bd0262203960 2023-03-29 03:56:39,860 DEBUG [regionserver/jenkins-hbase4:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.0, tune throughput to 50.00 MB/second 2023-03-29 03:56:40,112 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:40,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 285 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37666 deadline: 1680062260112, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:40,114 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-29 03:56:40,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] ipc.CallRunner(144): callId: 293 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37674 deadline: 1680062260114, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:40,195 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=712 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/C/099977b680da4486b27d522a18196d94 2023-03-29 03:56:40,199 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/A/2bfd75892d3a4e31bc32027b382accd1 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/2bfd75892d3a4e31bc32027b382accd1 2023-03-29 03:56:40,201 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/2bfd75892d3a4e31bc32027b382accd1, entries=200, sequenceid=712, filesize=32.3 K 2023-03-29 03:56:40,202 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/B/e28748f201a84445b832bd0262203960 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/e28748f201a84445b832bd0262203960 2023-03-29 03:56:40,204 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/e28748f201a84445b832bd0262203960, entries=150, sequenceid=712, filesize=11.9 K 2023-03-29 03:56:40,205 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/C/099977b680da4486b27d522a18196d94 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/099977b680da4486b27d522a18196d94 2023-03-29 03:56:40,207 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/099977b680da4486b27d522a18196d94, entries=150, sequenceid=712, filesize=11.9 K 2023-03-29 03:56:40,208 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=107.34 KB/109920 for 94b16ca160750195aa137348cdb80ba9 in 1245ms, sequenceid=712, compaction requested=true 2023-03-29 03:56:40,208 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 94b16ca160750195aa137348cdb80ba9: 2023-03-29 03:56:40,208 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:56:40,208 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:56:40,208 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-29 03:56:40,208 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-29 03:56:40,208 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:56:40,209 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 86540 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-29 03:56:40,209 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37886 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-29 03:56:40,209 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1912): 94b16ca160750195aa137348cdb80ba9/A is initiating minor compaction (all files) 2023-03-29 03:56:40,209 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1912): 94b16ca160750195aa137348cdb80ba9/B is initiating minor compaction (all files) 2023-03-29 03:56:40,209 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 94b16ca160750195aa137348cdb80ba9/A in TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:40,209 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 94b16ca160750195aa137348cdb80ba9/B in TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:40,209 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/6ec31d144bea47c18d0932199581adba, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/5ee3ad1521a14e81a1dae3a9d6f0701b, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/2bfd75892d3a4e31bc32027b382accd1] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp, totalSize=84.5 K 2023-03-29 03:56:40,209 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/f856925c8d034f8eb1b3d335aa2fa17b, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/5c8e4a38f4b448849d207223f33fb8a2, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/e28748f201a84445b832bd0262203960] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp, totalSize=37.0 K 2023-03-29 03:56:40,209 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 6ec31d144bea47c18d0932199581adba, keycount=150, bloomtype=ROW, size=26.8 K, encoding=NONE, compression=NONE, seqNum=679, earliestPutTs=1680062198040 2023-03-29 03:56:40,209 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting f856925c8d034f8eb1b3d335aa2fa17b, keycount=150, bloomtype=ROW, size=13.2 K, encoding=NONE, compression=NONE, seqNum=679, earliestPutTs=1680062198040 2023-03-29 03:56:40,210 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 5ee3ad1521a14e81a1dae3a9d6f0701b, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=698, earliestPutTs=1680062198732 2023-03-29 03:56:40,210 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting 5c8e4a38f4b448849d207223f33fb8a2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=698, earliestPutTs=1680062198732 2023-03-29 03:56:40,210 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 2bfd75892d3a4e31bc32027b382accd1, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=712, earliestPutTs=1680062198844 2023-03-29 03:56:40,210 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.Compactor(207): Compacting e28748f201a84445b832bd0262203960, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=712, earliestPutTs=1680062198844 2023-03-29 03:56:40,215 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] throttle.PressureAwareThroughputController(145): 94b16ca160750195aa137348cdb80ba9#B#compaction#715 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:56:40,217 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] throttle.PressureAwareThroughputController(145): 94b16ca160750195aa137348cdb80ba9#A#compaction#716 average throughput is 9.86 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:56:40,222 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/B/45116042b05141d8a6277943ebe47c56 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/45116042b05141d8a6277943ebe47c56 2023-03-29 03:56:40,226 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 94b16ca160750195aa137348cdb80ba9/B of 94b16ca160750195aa137348cdb80ba9 into 45116042b05141d8a6277943ebe47c56(size=13.3 K), total size for store is 13.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:56:40,226 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 94b16ca160750195aa137348cdb80ba9: 2023-03-29 03:56:40,226 INFO [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9., storeName=94b16ca160750195aa137348cdb80ba9/B, priority=13, startTime=1680062200208; duration=0sec 2023-03-29 03:56:40,226 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:56:40,226 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-03-29 03:56:40,226 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-03-29 03:56:40,227 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-03-29 03:56:40,227 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. because compaction request was cancelled 2023-03-29 03:56:40,230 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/A/d39c77ddc4f3415f960ef4daf450e6f6 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/d39c77ddc4f3415f960ef4daf450e6f6 2023-03-29 03:56:40,233 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 94b16ca160750195aa137348cdb80ba9/A of 94b16ca160750195aa137348cdb80ba9 into d39c77ddc4f3415f960ef4daf450e6f6(size=26.9 K), total size for store is 26.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:56:40,233 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 94b16ca160750195aa137348cdb80ba9: 2023-03-29 03:56:40,233 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9., storeName=94b16ca160750195aa137348cdb80ba9/A, priority=13, startTime=1680062200208; duration=0sec 2023-03-29 03:56:40,233 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:56:41,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35599] regionserver.HRegion(9158): Flush requested on 94b16ca160750195aa137348cdb80ba9 2023-03-29 03:56:41,117 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 94b16ca160750195aa137348cdb80ba9 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-29 03:56:41,117 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94b16ca160750195aa137348cdb80ba9, store=A 2023-03-29 03:56:41,117 DEBUG [Thread-4301] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2c53bf96 to 127.0.0.1:52103 2023-03-29 03:56:41,117 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:56:41,117 DEBUG [Thread-4301] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-29 03:56:41,117 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94b16ca160750195aa137348cdb80ba9, store=B 2023-03-29 03:56:41,117 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:56:41,117 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94b16ca160750195aa137348cdb80ba9, store=C 2023-03-29 03:56:41,117 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:56:41,120 DEBUG [Thread-4295] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x66b5b1f3 to 127.0.0.1:52103 2023-03-29 03:56:41,120 DEBUG [Thread-4295] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-29 03:56:41,131 DEBUG [Thread-4293] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x062b3caa to 127.0.0.1:52103 2023-03-29 03:56:41,131 DEBUG [Thread-4293] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-29 03:56:41,527 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032936885d881e34427883d258fac6b57f56 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032936885d881e34427883d258fac6b57f56 2023-03-29 03:56:41,931 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=737, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/A/d950cb46e2d74504ba633169be4a3e74 2023-03-29 03:56:41,938 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=737 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/B/0a706e94567548c4a2c9eff11ed2009f 2023-03-29 03:56:42,345 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=737 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/C/7feb213ca8a843bab19e0e4ab9f4d12f 2023-03-29 03:56:42,349 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/A/d950cb46e2d74504ba633169be4a3e74 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/d950cb46e2d74504ba633169be4a3e74 2023-03-29 03:56:42,352 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/d950cb46e2d74504ba633169be4a3e74, entries=150, sequenceid=737, filesize=25.4 K 2023-03-29 03:56:42,352 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/B/0a706e94567548c4a2c9eff11ed2009f as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/0a706e94567548c4a2c9eff11ed2009f 2023-03-29 03:56:42,355 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/0a706e94567548c4a2c9eff11ed2009f, entries=150, sequenceid=737, filesize=11.9 K 2023-03-29 03:56:42,355 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/C/7feb213ca8a843bab19e0e4ab9f4d12f as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/7feb213ca8a843bab19e0e4ab9f4d12f 2023-03-29 03:56:42,358 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/7feb213ca8a843bab19e0e4ab9f4d12f, entries=150, sequenceid=737, filesize=11.9 K 2023-03-29 03:56:42,358 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=13.42 KB/13740 for 94b16ca160750195aa137348cdb80ba9 in 1241ms, sequenceid=737, compaction requested=true 2023-03-29 03:56:42,359 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 94b16ca160750195aa137348cdb80ba9: 2023-03-29 03:56:42,359 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:56:42,359 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-03-29 03:56:42,359 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:56:42,359 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-03-29 03:56:42,359 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-29 03:56:42,359 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-03-29 03:56:42,359 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-03-29 03:56:42,359 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-03-29 03:56:42,359 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. because compaction request was cancelled 2023-03-29 03:56:42,359 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-03-29 03:56:42,360 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-29 03:56:42,360 DEBUG [RS:0;jenkins-hbase4:35599-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. because compaction request was cancelled 2023-03-29 03:56:42,360 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37920 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-29 03:56:42,360 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1912): 94b16ca160750195aa137348cdb80ba9/C is initiating minor compaction (all files) 2023-03-29 03:56:42,360 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 94b16ca160750195aa137348cdb80ba9/C in TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:42,360 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/49f3abbe870b48dd8be4abb9c9c02cfc, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/099977b680da4486b27d522a18196d94, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/7feb213ca8a843bab19e0e4ab9f4d12f] into tmpdir=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp, totalSize=37.0 K 2023-03-29 03:56:42,361 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 49f3abbe870b48dd8be4abb9c9c02cfc, keycount=150, bloomtype=ROW, size=13.2 K, encoding=NONE, compression=NONE, seqNum=698, earliestPutTs=1680062198732 2023-03-29 03:56:42,361 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 099977b680da4486b27d522a18196d94, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=712, earliestPutTs=1680062198844 2023-03-29 03:56:42,361 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] compactions.Compactor(207): Compacting 7feb213ca8a843bab19e0e4ab9f4d12f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=737, earliestPutTs=1680062198989 2023-03-29 03:56:42,366 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] throttle.PressureAwareThroughputController(145): 94b16ca160750195aa137348cdb80ba9#C#compaction#720 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-29 03:56:42,774 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/C/297cd7dd068e4df6b4941c8e4a296a0a as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/297cd7dd068e4df6b4941c8e4a296a0a 2023-03-29 03:56:42,777 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 94b16ca160750195aa137348cdb80ba9/C of 94b16ca160750195aa137348cdb80ba9 into 297cd7dd068e4df6b4941c8e4a296a0a(size=13.3 K), total size for store is 13.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-29 03:56:42,778 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 94b16ca160750195aa137348cdb80ba9: 2023-03-29 03:56:42,778 INFO [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9., storeName=94b16ca160750195aa137348cdb80ba9/C, priority=13, startTime=1680062202359; duration=0sec 2023-03-29 03:56:42,778 DEBUG [RS:0;jenkins-hbase4:35599-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-29 03:56:48,623 DEBUG [Thread-4299] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x25baee3e to 127.0.0.1:52103 2023-03-29 03:56:48,623 DEBUG [Thread-4299] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-29 03:56:48,642 DEBUG [Thread-4297] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x544a4215 to 127.0.0.1:52103 2023-03-29 03:56:48,642 DEBUG [Thread-4297] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-29 03:56:48,642 INFO [Listener at localhost/35027] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-03-29 03:56:48,642 INFO [Listener at localhost/35027] hbase.AcidGuaranteesTestTool(394): wrote 138 2023-03-29 03:56:48,642 INFO [Listener at localhost/35027] hbase.AcidGuaranteesTestTool(394): wrote 148 2023-03-29 03:56:48,642 INFO [Listener at localhost/35027] hbase.AcidGuaranteesTestTool(394): wrote 82 2023-03-29 03:56:48,642 INFO [Listener at localhost/35027] hbase.AcidGuaranteesTestTool(394): wrote 90 2023-03-29 03:56:48,642 INFO [Listener at localhost/35027] hbase.AcidGuaranteesTestTool(394): wrote 140 2023-03-29 03:56:48,642 INFO [Listener at localhost/35027] hbase.AcidGuaranteesTestTool(396): Readers: 2023-03-29 03:56:48,642 INFO [Listener at localhost/35027] hbase.AcidGuaranteesTestTool(398): read 4792 2023-03-29 03:56:48,642 INFO [Listener at localhost/35027] hbase.AcidGuaranteesTestTool(398): read 4782 2023-03-29 03:56:48,642 INFO [Listener at localhost/35027] hbase.AcidGuaranteesTestTool(398): read 4637 2023-03-29 03:56:48,642 INFO [Listener at localhost/35027] hbase.AcidGuaranteesTestTool(398): read 4789 2023-03-29 03:56:48,642 INFO [Listener at localhost/35027] hbase.AcidGuaranteesTestTool(398): read 4780 2023-03-29 03:56:48,642 INFO [Listener at localhost/35027] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-03-29 03:56:48,642 INFO [Listener at localhost/35027] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-03-29 03:56:48,643 DEBUG [Listener at localhost/35027] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x59185ba7 to 127.0.0.1:52103 2023-03-29 03:56:48,643 DEBUG [Listener at localhost/35027] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-29 03:56:48,643 INFO [Listener at localhost/35027] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-03-29 03:56:48,644 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-03-29 03:56:48,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure2.ProcedureExecutor(1029): Stored pid=62, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-03-29 03:56:48,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-03-29 03:56:48,646 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680062208646"}]},"ts":"1680062208646"} 2023-03-29 03:56:48,647 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-03-29 03:56:48,652 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-03-29 03:56:48,652 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=63, ppid=62, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=94b16ca160750195aa137348cdb80ba9, UNASSIGN}] 2023-03-29 03:56:48,654 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=63, ppid=62, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=94b16ca160750195aa137348cdb80ba9, UNASSIGN 2023-03-29 03:56:48,654 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=63 updating hbase:meta row=94b16ca160750195aa137348cdb80ba9, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:48,654 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680062208654"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680062208654"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680062208654"}]},"ts":"1680062208654"} 2023-03-29 03:56:48,655 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=64, ppid=63, state=RUNNABLE; CloseRegionProcedure 94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534}] 2023-03-29 03:56:48,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-03-29 03:56:48,807 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 94b16ca160750195aa137348cdb80ba9 2023-03-29 03:56:48,807 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 94b16ca160750195aa137348cdb80ba9, disabling compactions & flushes 2023-03-29 03:56:48,807 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:48,807 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:48,807 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. after waiting 0 ms 2023-03-29 03:56:48,807 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:48,807 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 94b16ca160750195aa137348cdb80ba9 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-03-29 03:56:48,807 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94b16ca160750195aa137348cdb80ba9, store=A 2023-03-29 03:56:48,807 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:56:48,807 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94b16ca160750195aa137348cdb80ba9, store=B 2023-03-29 03:56:48,807 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:56:48,807 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 94b16ca160750195aa137348cdb80ba9, store=C 2023-03-29 03:56:48,807 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-29 03:56:48,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-03-29 03:56:49,217 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303294c3cadbb5d704bfd8537c9497f039336 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303294c3cadbb5d704bfd8537c9497f039336 2023-03-29 03:56:49,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-03-29 03:56:49,622 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=745, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/A/91e2346552f341af9aabad616a4a6349 2023-03-29 03:56:49,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-03-29 03:56:50,030 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=745 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/B/8e788225d4fc46f59e082824254224b9 2023-03-29 03:56:50,438 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=745 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/C/ac98ebe871c248619215c94276b44641 2023-03-29 03:56:50,441 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/A/91e2346552f341af9aabad616a4a6349 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/91e2346552f341af9aabad616a4a6349 2023-03-29 03:56:50,444 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/91e2346552f341af9aabad616a4a6349, entries=150, sequenceid=745, filesize=25.4 K 2023-03-29 03:56:50,444 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/B/8e788225d4fc46f59e082824254224b9 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/8e788225d4fc46f59e082824254224b9 2023-03-29 03:56:50,447 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/8e788225d4fc46f59e082824254224b9, entries=150, sequenceid=745, filesize=11.9 K 2023-03-29 03:56:50,448 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/.tmp/C/ac98ebe871c248619215c94276b44641 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/ac98ebe871c248619215c94276b44641 2023-03-29 03:56:50,451 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/ac98ebe871c248619215c94276b44641, entries=150, sequenceid=745, filesize=11.9 K 2023-03-29 03:56:50,451 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for 94b16ca160750195aa137348cdb80ba9 in 1644ms, sequenceid=745, compaction requested=true 2023-03-29 03:56:50,453 DEBUG [StoreCloser-TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/f9f67b3b0d4b4ba3a97ede9e12e36d77, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/8725b6616cf24accad5f453fe46e4ed2, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/183379056af8462fbc92a6b88d79b344, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/d414b1190d624823baac2744acbe5edf, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/88fdd8f90adb43a39a41aeb6f5d59e69, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/2c7bb4c01e1346e7b210bbc1a227a40f, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/5d565495fa2048cfb38c6fcb5a7afb0d, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/4b174ccdd8864f3e9f5577c828034cdf, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/54df16ea5a224f04869e997b7003f9fb, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/84cb900ce4b94ae189ed1e7099c0124e, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/629361042e7d4e15996be65d1b80128c, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/f40864718506408c8d813380f86cead0, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/4bf75005213b4e6b88cafa073e2e8d13, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/fd56c068636b41b88a44c4782d008ef1, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/2bb3bb5a36a34ebda2648f98e6f9729e, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/c730b09e1c5e4d9a8983539574402633, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/11ce9d02c9c941a4a9277d2ef2992926, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/a4b76454c8f14de0bcbac98e0382d22b, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/aee92c8248ce40e3a06f1cd5c14af621, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/e4ef63ec149f4296a2037aa8e4069b30, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/51570dbb8b0d45fa87eb33fd8e50f2ac, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/ce2387e240424455b65afa1be80c1acf, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/57ac31b6f6154938a870c668c5c71262, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/09bfad8a80fc4065acbe5fab950215ee, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/6fcfc2d120014d98a52272fb58aa528e, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/dbadf17b79be41ce95bf89690cae8ce3, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/98fb35f39d80418eaf39b148fd06851e, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/278e7ca258b84677bc787eef8a8b4db1, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/0f88f7db9db248a6951e27276f7e1c39, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/909e9c66618b4d10a32f16171ae8b859, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/08fa46538e654919906bb95a7b5793b3, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/5d846473c6394ba2808bc0da9d78ba89, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/2e676da0a5854fc7b82830c3f7a54d45, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/809da56654e443bd9cc70390e7feb12f, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/1334983c9f60479e814150fcbd8422fc, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/d4f5c089004942d39f09ef4691b549af, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/5fe313a6ed2b4b5a96df42c4312fbbdb, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/54f9e76b5c004f91ae3defa3cae79fff, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/3d0083d5c2fb42a4b3440162bcb97267, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/0d091bd580d34876845e40599a3ed49a, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/6ec31d144bea47c18d0932199581adba, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/5ee3ad1521a14e81a1dae3a9d6f0701b, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/2bfd75892d3a4e31bc32027b382accd1] to archive 2023-03-29 03:56:50,454 DEBUG [StoreCloser-TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-29 03:56:50,455 DEBUG [StoreCloser-TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/f9f67b3b0d4b4ba3a97ede9e12e36d77 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/f9f67b3b0d4b4ba3a97ede9e12e36d77 2023-03-29 03:56:50,456 DEBUG [StoreCloser-TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/8725b6616cf24accad5f453fe46e4ed2 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/8725b6616cf24accad5f453fe46e4ed2 2023-03-29 03:56:50,457 DEBUG [StoreCloser-TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/183379056af8462fbc92a6b88d79b344 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/183379056af8462fbc92a6b88d79b344 2023-03-29 03:56:50,458 DEBUG [StoreCloser-TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/d414b1190d624823baac2744acbe5edf to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/d414b1190d624823baac2744acbe5edf 2023-03-29 03:56:50,459 DEBUG [StoreCloser-TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/88fdd8f90adb43a39a41aeb6f5d59e69 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/88fdd8f90adb43a39a41aeb6f5d59e69 2023-03-29 03:56:50,460 DEBUG [StoreCloser-TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/2c7bb4c01e1346e7b210bbc1a227a40f to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/2c7bb4c01e1346e7b210bbc1a227a40f 2023-03-29 03:56:50,461 DEBUG [StoreCloser-TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/5d565495fa2048cfb38c6fcb5a7afb0d to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/5d565495fa2048cfb38c6fcb5a7afb0d 2023-03-29 03:56:50,462 DEBUG [StoreCloser-TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/4b174ccdd8864f3e9f5577c828034cdf to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/4b174ccdd8864f3e9f5577c828034cdf 2023-03-29 03:56:50,462 DEBUG [StoreCloser-TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/54df16ea5a224f04869e997b7003f9fb to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/54df16ea5a224f04869e997b7003f9fb 2023-03-29 03:56:50,463 DEBUG [StoreCloser-TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/84cb900ce4b94ae189ed1e7099c0124e to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/84cb900ce4b94ae189ed1e7099c0124e 2023-03-29 03:56:50,464 DEBUG [StoreCloser-TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/629361042e7d4e15996be65d1b80128c to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/629361042e7d4e15996be65d1b80128c 2023-03-29 03:56:50,465 DEBUG [StoreCloser-TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/f40864718506408c8d813380f86cead0 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/f40864718506408c8d813380f86cead0 2023-03-29 03:56:50,466 DEBUG [StoreCloser-TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/4bf75005213b4e6b88cafa073e2e8d13 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/4bf75005213b4e6b88cafa073e2e8d13 2023-03-29 03:56:50,467 DEBUG [StoreCloser-TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/fd56c068636b41b88a44c4782d008ef1 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/fd56c068636b41b88a44c4782d008ef1 2023-03-29 03:56:50,468 DEBUG [StoreCloser-TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/2bb3bb5a36a34ebda2648f98e6f9729e to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/2bb3bb5a36a34ebda2648f98e6f9729e 2023-03-29 03:56:50,469 DEBUG [StoreCloser-TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/c730b09e1c5e4d9a8983539574402633 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/c730b09e1c5e4d9a8983539574402633 2023-03-29 03:56:50,469 DEBUG [StoreCloser-TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/11ce9d02c9c941a4a9277d2ef2992926 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/11ce9d02c9c941a4a9277d2ef2992926 2023-03-29 03:56:50,470 DEBUG [StoreCloser-TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/a4b76454c8f14de0bcbac98e0382d22b to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/a4b76454c8f14de0bcbac98e0382d22b 2023-03-29 03:56:50,471 DEBUG [StoreCloser-TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/aee92c8248ce40e3a06f1cd5c14af621 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/aee92c8248ce40e3a06f1cd5c14af621 2023-03-29 03:56:50,472 DEBUG [StoreCloser-TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/e4ef63ec149f4296a2037aa8e4069b30 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/e4ef63ec149f4296a2037aa8e4069b30 2023-03-29 03:56:50,472 DEBUG [StoreCloser-TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/51570dbb8b0d45fa87eb33fd8e50f2ac to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/51570dbb8b0d45fa87eb33fd8e50f2ac 2023-03-29 03:56:50,473 DEBUG [StoreCloser-TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/ce2387e240424455b65afa1be80c1acf to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/ce2387e240424455b65afa1be80c1acf 2023-03-29 03:56:50,474 DEBUG [StoreCloser-TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/57ac31b6f6154938a870c668c5c71262 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/57ac31b6f6154938a870c668c5c71262 2023-03-29 03:56:50,475 DEBUG [StoreCloser-TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/09bfad8a80fc4065acbe5fab950215ee to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/09bfad8a80fc4065acbe5fab950215ee 2023-03-29 03:56:50,476 DEBUG [StoreCloser-TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/6fcfc2d120014d98a52272fb58aa528e to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/6fcfc2d120014d98a52272fb58aa528e 2023-03-29 03:56:50,476 DEBUG [StoreCloser-TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/dbadf17b79be41ce95bf89690cae8ce3 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/dbadf17b79be41ce95bf89690cae8ce3 2023-03-29 03:56:50,477 DEBUG [StoreCloser-TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/98fb35f39d80418eaf39b148fd06851e to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/98fb35f39d80418eaf39b148fd06851e 2023-03-29 03:56:50,478 DEBUG [StoreCloser-TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/278e7ca258b84677bc787eef8a8b4db1 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/278e7ca258b84677bc787eef8a8b4db1 2023-03-29 03:56:50,479 DEBUG [StoreCloser-TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/0f88f7db9db248a6951e27276f7e1c39 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/0f88f7db9db248a6951e27276f7e1c39 2023-03-29 03:56:50,480 DEBUG [StoreCloser-TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/909e9c66618b4d10a32f16171ae8b859 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/909e9c66618b4d10a32f16171ae8b859 2023-03-29 03:56:50,480 DEBUG [StoreCloser-TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/08fa46538e654919906bb95a7b5793b3 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/08fa46538e654919906bb95a7b5793b3 2023-03-29 03:56:50,481 DEBUG [StoreCloser-TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/5d846473c6394ba2808bc0da9d78ba89 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/5d846473c6394ba2808bc0da9d78ba89 2023-03-29 03:56:50,482 DEBUG [StoreCloser-TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/2e676da0a5854fc7b82830c3f7a54d45 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/2e676da0a5854fc7b82830c3f7a54d45 2023-03-29 03:56:50,483 DEBUG [StoreCloser-TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/809da56654e443bd9cc70390e7feb12f to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/809da56654e443bd9cc70390e7feb12f 2023-03-29 03:56:50,484 DEBUG [StoreCloser-TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/1334983c9f60479e814150fcbd8422fc to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/1334983c9f60479e814150fcbd8422fc 2023-03-29 03:56:50,485 DEBUG [StoreCloser-TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/d4f5c089004942d39f09ef4691b549af to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/d4f5c089004942d39f09ef4691b549af 2023-03-29 03:56:50,486 DEBUG [StoreCloser-TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/5fe313a6ed2b4b5a96df42c4312fbbdb to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/5fe313a6ed2b4b5a96df42c4312fbbdb 2023-03-29 03:56:50,487 DEBUG [StoreCloser-TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/54f9e76b5c004f91ae3defa3cae79fff to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/54f9e76b5c004f91ae3defa3cae79fff 2023-03-29 03:56:50,488 DEBUG [StoreCloser-TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/3d0083d5c2fb42a4b3440162bcb97267 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/3d0083d5c2fb42a4b3440162bcb97267 2023-03-29 03:56:50,489 DEBUG [StoreCloser-TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/0d091bd580d34876845e40599a3ed49a to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/0d091bd580d34876845e40599a3ed49a 2023-03-29 03:56:50,492 DEBUG [StoreCloser-TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/6ec31d144bea47c18d0932199581adba to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/6ec31d144bea47c18d0932199581adba 2023-03-29 03:56:50,493 DEBUG [StoreCloser-TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/5ee3ad1521a14e81a1dae3a9d6f0701b to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/5ee3ad1521a14e81a1dae3a9d6f0701b 2023-03-29 03:56:50,494 DEBUG [StoreCloser-TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/2bfd75892d3a4e31bc32027b382accd1 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/2bfd75892d3a4e31bc32027b382accd1 2023-03-29 03:56:50,497 DEBUG [StoreCloser-TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/7e4484834edd46fe92767797e5512b77, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/284d397ea2b44c92ab4085b6b0e584e3, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/344bc1207b904bf6a57af2e3a1f2c3cd, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/cd4aef3dc6414c10bf2b93115d929229, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/621c55fa87d34fbfaf88b0cd59718562, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/9ee0db13a4d54259b15727e487810f0d, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/aa0f4a9b2728441c89a9c8fe10c5efc7, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/1676009485bb4324b35181b980cedc7d, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/59b2356b806a45a58edc24610ac15efd, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/84a4345f266c4fa599d5112825a964d3, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/37682f23d96a415593111dc008377dcd, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/a9e314fab9b8438b800d7b027f45c676, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/32523715f3fb4f3694681eb8024aa11d, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/877e2162b8a4487e8ef12c4da960d6fd, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/31c17bc54c8c4265bfae30cf3621e3fe, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/14f7ea675c654119ad69b22f47ed2a2d, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/782ec38b1b984fbeb2d1995ba4db5b8d, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/08961dda88994f33b09ce83318cf63c0, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/abfa254bacad4eeaac55353cf184f95a, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/6c3d3f961160426aa3de2574eb4d22d6, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/0993640b669a4f09ae3a0cabfdc08c06, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/378a1f6a9b534948be4de07e90e13e3f, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/c17d39257d914de4aad2152a7a258cb6, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/97a7133da9404743a42ceeb0b60b5fb2, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/19c6bd442a024904a38a99820d2ef985, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/7add7a28162f4da986732d342d667dd6, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/4f9daa334e1d4ee59e584c0283d13ed7, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/7255786c3ec0489482b9262ae610a3dd, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/91c27693585c4f9b81f20b4c3b8b326a, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/695af9f547f841a0bd277152911152e8, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/ee8c886a3e4c41fa815e48a468cc625c, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/92b020720e494364add1a6fa86ceeb7f, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/72766e198d6f4e288875beaaa949c941, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/747b3a2693804be7b44985c12b01b66d, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/086af0428d4c41dca7ae6ae02ed8e0cc, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/e7765708c0d24c55bb0cc39c85057bd4, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/1240308bce954b3fae49e93edf24cbc3, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/4c49bd9cc7b947c5bbd96a75e8196a72, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/f856925c8d034f8eb1b3d335aa2fa17b, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/acbc9e9359cb4536b40a0957087c8beb, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/5c8e4a38f4b448849d207223f33fb8a2, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/e28748f201a84445b832bd0262203960] to archive 2023-03-29 03:56:50,498 DEBUG [StoreCloser-TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-29 03:56:50,499 DEBUG [StoreCloser-TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/7e4484834edd46fe92767797e5512b77 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/7e4484834edd46fe92767797e5512b77 2023-03-29 03:56:50,500 DEBUG [StoreCloser-TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/284d397ea2b44c92ab4085b6b0e584e3 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/284d397ea2b44c92ab4085b6b0e584e3 2023-03-29 03:56:50,501 DEBUG [StoreCloser-TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/344bc1207b904bf6a57af2e3a1f2c3cd to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/344bc1207b904bf6a57af2e3a1f2c3cd 2023-03-29 03:56:50,502 DEBUG [StoreCloser-TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/cd4aef3dc6414c10bf2b93115d929229 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/cd4aef3dc6414c10bf2b93115d929229 2023-03-29 03:56:50,502 DEBUG [StoreCloser-TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/621c55fa87d34fbfaf88b0cd59718562 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/621c55fa87d34fbfaf88b0cd59718562 2023-03-29 03:56:50,503 DEBUG [StoreCloser-TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/9ee0db13a4d54259b15727e487810f0d to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/9ee0db13a4d54259b15727e487810f0d 2023-03-29 03:56:50,504 DEBUG [StoreCloser-TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/aa0f4a9b2728441c89a9c8fe10c5efc7 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/aa0f4a9b2728441c89a9c8fe10c5efc7 2023-03-29 03:56:50,505 DEBUG [StoreCloser-TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/1676009485bb4324b35181b980cedc7d to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/1676009485bb4324b35181b980cedc7d 2023-03-29 03:56:50,506 DEBUG [StoreCloser-TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/59b2356b806a45a58edc24610ac15efd to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/59b2356b806a45a58edc24610ac15efd 2023-03-29 03:56:50,506 DEBUG [StoreCloser-TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/84a4345f266c4fa599d5112825a964d3 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/84a4345f266c4fa599d5112825a964d3 2023-03-29 03:56:50,507 DEBUG [StoreCloser-TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/37682f23d96a415593111dc008377dcd to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/37682f23d96a415593111dc008377dcd 2023-03-29 03:56:50,508 DEBUG [StoreCloser-TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/a9e314fab9b8438b800d7b027f45c676 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/a9e314fab9b8438b800d7b027f45c676 2023-03-29 03:56:50,508 DEBUG [StoreCloser-TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/32523715f3fb4f3694681eb8024aa11d to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/32523715f3fb4f3694681eb8024aa11d 2023-03-29 03:56:50,509 DEBUG [StoreCloser-TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/877e2162b8a4487e8ef12c4da960d6fd to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/877e2162b8a4487e8ef12c4da960d6fd 2023-03-29 03:56:50,510 DEBUG [StoreCloser-TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/31c17bc54c8c4265bfae30cf3621e3fe to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/31c17bc54c8c4265bfae30cf3621e3fe 2023-03-29 03:56:50,511 DEBUG [StoreCloser-TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/14f7ea675c654119ad69b22f47ed2a2d to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/14f7ea675c654119ad69b22f47ed2a2d 2023-03-29 03:56:50,511 DEBUG [StoreCloser-TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/782ec38b1b984fbeb2d1995ba4db5b8d to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/782ec38b1b984fbeb2d1995ba4db5b8d 2023-03-29 03:56:50,512 DEBUG [StoreCloser-TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/08961dda88994f33b09ce83318cf63c0 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/08961dda88994f33b09ce83318cf63c0 2023-03-29 03:56:50,513 DEBUG [StoreCloser-TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/abfa254bacad4eeaac55353cf184f95a to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/abfa254bacad4eeaac55353cf184f95a 2023-03-29 03:56:50,513 DEBUG [StoreCloser-TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/6c3d3f961160426aa3de2574eb4d22d6 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/6c3d3f961160426aa3de2574eb4d22d6 2023-03-29 03:56:50,514 DEBUG [StoreCloser-TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/0993640b669a4f09ae3a0cabfdc08c06 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/0993640b669a4f09ae3a0cabfdc08c06 2023-03-29 03:56:50,515 DEBUG [StoreCloser-TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/378a1f6a9b534948be4de07e90e13e3f to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/378a1f6a9b534948be4de07e90e13e3f 2023-03-29 03:56:50,516 DEBUG [StoreCloser-TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/c17d39257d914de4aad2152a7a258cb6 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/c17d39257d914de4aad2152a7a258cb6 2023-03-29 03:56:50,516 DEBUG [StoreCloser-TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/97a7133da9404743a42ceeb0b60b5fb2 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/97a7133da9404743a42ceeb0b60b5fb2 2023-03-29 03:56:50,517 DEBUG [StoreCloser-TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/19c6bd442a024904a38a99820d2ef985 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/19c6bd442a024904a38a99820d2ef985 2023-03-29 03:56:50,518 DEBUG [StoreCloser-TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/7add7a28162f4da986732d342d667dd6 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/7add7a28162f4da986732d342d667dd6 2023-03-29 03:56:50,518 DEBUG [StoreCloser-TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/4f9daa334e1d4ee59e584c0283d13ed7 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/4f9daa334e1d4ee59e584c0283d13ed7 2023-03-29 03:56:50,519 DEBUG [StoreCloser-TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/7255786c3ec0489482b9262ae610a3dd to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/7255786c3ec0489482b9262ae610a3dd 2023-03-29 03:56:50,520 DEBUG [StoreCloser-TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/91c27693585c4f9b81f20b4c3b8b326a to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/91c27693585c4f9b81f20b4c3b8b326a 2023-03-29 03:56:50,521 DEBUG [StoreCloser-TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/695af9f547f841a0bd277152911152e8 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/695af9f547f841a0bd277152911152e8 2023-03-29 03:56:50,521 DEBUG [StoreCloser-TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/ee8c886a3e4c41fa815e48a468cc625c to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/ee8c886a3e4c41fa815e48a468cc625c 2023-03-29 03:56:50,522 DEBUG [StoreCloser-TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/92b020720e494364add1a6fa86ceeb7f to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/92b020720e494364add1a6fa86ceeb7f 2023-03-29 03:56:50,523 DEBUG [StoreCloser-TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/72766e198d6f4e288875beaaa949c941 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/72766e198d6f4e288875beaaa949c941 2023-03-29 03:56:50,524 DEBUG [StoreCloser-TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/747b3a2693804be7b44985c12b01b66d to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/747b3a2693804be7b44985c12b01b66d 2023-03-29 03:56:50,524 DEBUG [StoreCloser-TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/086af0428d4c41dca7ae6ae02ed8e0cc to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/086af0428d4c41dca7ae6ae02ed8e0cc 2023-03-29 03:56:50,525 DEBUG [StoreCloser-TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/e7765708c0d24c55bb0cc39c85057bd4 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/e7765708c0d24c55bb0cc39c85057bd4 2023-03-29 03:56:50,526 DEBUG [StoreCloser-TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/1240308bce954b3fae49e93edf24cbc3 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/1240308bce954b3fae49e93edf24cbc3 2023-03-29 03:56:50,526 DEBUG [StoreCloser-TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/4c49bd9cc7b947c5bbd96a75e8196a72 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/4c49bd9cc7b947c5bbd96a75e8196a72 2023-03-29 03:56:50,527 DEBUG [StoreCloser-TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/f856925c8d034f8eb1b3d335aa2fa17b to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/f856925c8d034f8eb1b3d335aa2fa17b 2023-03-29 03:56:50,528 DEBUG [StoreCloser-TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/acbc9e9359cb4536b40a0957087c8beb to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/acbc9e9359cb4536b40a0957087c8beb 2023-03-29 03:56:50,528 DEBUG [StoreCloser-TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/5c8e4a38f4b448849d207223f33fb8a2 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/5c8e4a38f4b448849d207223f33fb8a2 2023-03-29 03:56:50,529 DEBUG [StoreCloser-TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/e28748f201a84445b832bd0262203960 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/e28748f201a84445b832bd0262203960 2023-03-29 03:56:50,532 DEBUG [StoreCloser-TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/98d37c2abeea4853bad36ac1088dc8d6, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/a0bd07e926d5475c8fbef239de91c279, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/bc6336b3978840f4ab53bb5d285b6b7d, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/f2ca8378cb984ac18483d4740747566b, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/89d89c0b2327423b9a082530d2a61350, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/67394449c9d94d699b3209ea6ff531ce, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/f9497ac0271648b88593fd5c961819b6, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/e72e7307fb044791a44923488323e1d7, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/52aaa9b60345448fb005082290b00994, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/fcbdadeb4cc34a7fbe2ca17050241f80, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/d284df99f2824152baff1fae57af725a, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/b0e1d19cd0614a2d942e5b1301fbceb0, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/ca25372fb32b48c08a5c970a1d76c719, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/bba0413ad9e041cf9e98da70608d23c3, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/de05caeb70ba4557bf484790fe90404b, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/40df171f6f5a4e4daa24a80563045fb3, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/5928c0f34fb54229b579133bed9e3881, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/9997ca2063044d3b89fedb3f63e599c4, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/22ffbecf3e004da5aada22a0162d1473, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/50a0006b926e457fa286a13cb3d0ff24, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/47647c64df1d43358d3ae25f251add7d, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/ab929623b702435abdee9940aa0226df, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/b1d9e9da33bb48e29b1368ca97f2d542, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/dfb24adb4047429ebdbdc3626fec41b5, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/2360ffa733d54906bc0b03bfeb3e0245, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/cfd96a66e048477db7c93f407268f4a7, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/8d6666c36f614d7f966944a505b1da12, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/1d1bc128614643e487e34b125b7b18a8, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/93c93226f9fb473097488adce0ec58a4, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/92807488dc5f442ba8f7eb9bf9a76336, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/93ad53cb25844d3ea023735a413b020a, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/a4671f180dc04808af5c7636ad4ea19c, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/4c13dbbe82664e82b2b413594a35af19, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/713000b021b24cd4b0c15de1446c9c1a, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/995bdb7bc9b84e8e97e59cc671d1b755, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/05992b62470540778f3858dba335af3b, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/00490b2eb48a4d498cf6224cfd1ebb49, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/c848fe1465024bf390f67d59d90b3548, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/9035f3877f4a401b82e2ab0cb928a270, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/49f3abbe870b48dd8be4abb9c9c02cfc, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/2fae292ba5464616ad301a39e3bb3cb6, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/099977b680da4486b27d522a18196d94, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/7feb213ca8a843bab19e0e4ab9f4d12f] to archive 2023-03-29 03:56:50,532 DEBUG [StoreCloser-TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-29 03:56:50,534 DEBUG [StoreCloser-TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/98d37c2abeea4853bad36ac1088dc8d6 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/98d37c2abeea4853bad36ac1088dc8d6 2023-03-29 03:56:50,534 DEBUG [StoreCloser-TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/a0bd07e926d5475c8fbef239de91c279 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/a0bd07e926d5475c8fbef239de91c279 2023-03-29 03:56:50,535 DEBUG [StoreCloser-TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/bc6336b3978840f4ab53bb5d285b6b7d to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/bc6336b3978840f4ab53bb5d285b6b7d 2023-03-29 03:56:50,536 DEBUG [StoreCloser-TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/f2ca8378cb984ac18483d4740747566b to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/f2ca8378cb984ac18483d4740747566b 2023-03-29 03:56:50,536 DEBUG [StoreCloser-TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/89d89c0b2327423b9a082530d2a61350 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/89d89c0b2327423b9a082530d2a61350 2023-03-29 03:56:50,537 DEBUG [StoreCloser-TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/67394449c9d94d699b3209ea6ff531ce to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/67394449c9d94d699b3209ea6ff531ce 2023-03-29 03:56:50,538 DEBUG [StoreCloser-TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/f9497ac0271648b88593fd5c961819b6 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/f9497ac0271648b88593fd5c961819b6 2023-03-29 03:56:50,539 DEBUG [StoreCloser-TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/e72e7307fb044791a44923488323e1d7 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/e72e7307fb044791a44923488323e1d7 2023-03-29 03:56:50,539 DEBUG [StoreCloser-TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/52aaa9b60345448fb005082290b00994 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/52aaa9b60345448fb005082290b00994 2023-03-29 03:56:50,540 DEBUG [StoreCloser-TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/fcbdadeb4cc34a7fbe2ca17050241f80 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/fcbdadeb4cc34a7fbe2ca17050241f80 2023-03-29 03:56:50,541 DEBUG [StoreCloser-TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/d284df99f2824152baff1fae57af725a to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/d284df99f2824152baff1fae57af725a 2023-03-29 03:56:50,541 DEBUG [StoreCloser-TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/b0e1d19cd0614a2d942e5b1301fbceb0 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/b0e1d19cd0614a2d942e5b1301fbceb0 2023-03-29 03:56:50,542 DEBUG [StoreCloser-TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/ca25372fb32b48c08a5c970a1d76c719 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/ca25372fb32b48c08a5c970a1d76c719 2023-03-29 03:56:50,543 DEBUG [StoreCloser-TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/bba0413ad9e041cf9e98da70608d23c3 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/bba0413ad9e041cf9e98da70608d23c3 2023-03-29 03:56:50,544 DEBUG [StoreCloser-TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/de05caeb70ba4557bf484790fe90404b to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/de05caeb70ba4557bf484790fe90404b 2023-03-29 03:56:50,544 DEBUG [StoreCloser-TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/40df171f6f5a4e4daa24a80563045fb3 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/40df171f6f5a4e4daa24a80563045fb3 2023-03-29 03:56:50,545 DEBUG [StoreCloser-TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/5928c0f34fb54229b579133bed9e3881 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/5928c0f34fb54229b579133bed9e3881 2023-03-29 03:56:50,546 DEBUG [StoreCloser-TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/9997ca2063044d3b89fedb3f63e599c4 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/9997ca2063044d3b89fedb3f63e599c4 2023-03-29 03:56:50,546 DEBUG [StoreCloser-TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/22ffbecf3e004da5aada22a0162d1473 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/22ffbecf3e004da5aada22a0162d1473 2023-03-29 03:56:50,547 DEBUG [StoreCloser-TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/50a0006b926e457fa286a13cb3d0ff24 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/50a0006b926e457fa286a13cb3d0ff24 2023-03-29 03:56:50,548 DEBUG [StoreCloser-TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/47647c64df1d43358d3ae25f251add7d to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/47647c64df1d43358d3ae25f251add7d 2023-03-29 03:56:50,549 DEBUG [StoreCloser-TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/ab929623b702435abdee9940aa0226df to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/ab929623b702435abdee9940aa0226df 2023-03-29 03:56:50,549 DEBUG [StoreCloser-TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/b1d9e9da33bb48e29b1368ca97f2d542 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/b1d9e9da33bb48e29b1368ca97f2d542 2023-03-29 03:56:50,550 DEBUG [StoreCloser-TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/dfb24adb4047429ebdbdc3626fec41b5 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/dfb24adb4047429ebdbdc3626fec41b5 2023-03-29 03:56:50,551 DEBUG [StoreCloser-TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/2360ffa733d54906bc0b03bfeb3e0245 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/2360ffa733d54906bc0b03bfeb3e0245 2023-03-29 03:56:50,551 DEBUG [StoreCloser-TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/cfd96a66e048477db7c93f407268f4a7 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/cfd96a66e048477db7c93f407268f4a7 2023-03-29 03:56:50,552 DEBUG [StoreCloser-TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/8d6666c36f614d7f966944a505b1da12 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/8d6666c36f614d7f966944a505b1da12 2023-03-29 03:56:50,553 DEBUG [StoreCloser-TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/1d1bc128614643e487e34b125b7b18a8 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/1d1bc128614643e487e34b125b7b18a8 2023-03-29 03:56:50,554 DEBUG [StoreCloser-TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/93c93226f9fb473097488adce0ec58a4 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/93c93226f9fb473097488adce0ec58a4 2023-03-29 03:56:50,554 DEBUG [StoreCloser-TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/92807488dc5f442ba8f7eb9bf9a76336 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/92807488dc5f442ba8f7eb9bf9a76336 2023-03-29 03:56:50,555 DEBUG [StoreCloser-TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/93ad53cb25844d3ea023735a413b020a to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/93ad53cb25844d3ea023735a413b020a 2023-03-29 03:56:50,556 DEBUG [StoreCloser-TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/a4671f180dc04808af5c7636ad4ea19c to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/a4671f180dc04808af5c7636ad4ea19c 2023-03-29 03:56:50,557 DEBUG [StoreCloser-TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/4c13dbbe82664e82b2b413594a35af19 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/4c13dbbe82664e82b2b413594a35af19 2023-03-29 03:56:50,557 DEBUG [StoreCloser-TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/713000b021b24cd4b0c15de1446c9c1a to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/713000b021b24cd4b0c15de1446c9c1a 2023-03-29 03:56:50,558 DEBUG [StoreCloser-TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/995bdb7bc9b84e8e97e59cc671d1b755 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/995bdb7bc9b84e8e97e59cc671d1b755 2023-03-29 03:56:50,559 DEBUG [StoreCloser-TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/05992b62470540778f3858dba335af3b to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/05992b62470540778f3858dba335af3b 2023-03-29 03:56:50,560 DEBUG [StoreCloser-TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/00490b2eb48a4d498cf6224cfd1ebb49 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/00490b2eb48a4d498cf6224cfd1ebb49 2023-03-29 03:56:50,560 DEBUG [StoreCloser-TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/c848fe1465024bf390f67d59d90b3548 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/c848fe1465024bf390f67d59d90b3548 2023-03-29 03:56:50,561 DEBUG [StoreCloser-TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/9035f3877f4a401b82e2ab0cb928a270 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/9035f3877f4a401b82e2ab0cb928a270 2023-03-29 03:56:50,562 DEBUG [StoreCloser-TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/49f3abbe870b48dd8be4abb9c9c02cfc to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/49f3abbe870b48dd8be4abb9c9c02cfc 2023-03-29 03:56:50,563 DEBUG [StoreCloser-TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/2fae292ba5464616ad301a39e3bb3cb6 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/2fae292ba5464616ad301a39e3bb3cb6 2023-03-29 03:56:50,564 DEBUG [StoreCloser-TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/099977b680da4486b27d522a18196d94 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/099977b680da4486b27d522a18196d94 2023-03-29 03:56:50,565 DEBUG [StoreCloser-TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/7feb213ca8a843bab19e0e4ab9f4d12f to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/7feb213ca8a843bab19e0e4ab9f4d12f 2023-03-29 03:56:50,568 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/recovered.edits/748.seqid, newMaxSeqId=748, maxSeqId=4 2023-03-29 03:56:50,569 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9. 2023-03-29 03:56:50,569 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 94b16ca160750195aa137348cdb80ba9: 2023-03-29 03:56:50,570 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 94b16ca160750195aa137348cdb80ba9 2023-03-29 03:56:50,571 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=63 updating hbase:meta row=94b16ca160750195aa137348cdb80ba9, regionState=CLOSED 2023-03-29 03:56:50,571 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680062210571"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680062210571"}]},"ts":"1680062210571"} 2023-03-29 03:56:50,573 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=64, resume processing ppid=63 2023-03-29 03:56:50,573 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=64, ppid=63, state=SUCCESS; CloseRegionProcedure 94b16ca160750195aa137348cdb80ba9, server=jenkins-hbase4.apache.org,35599,1680062018534 in 1.9170 sec 2023-03-29 03:56:50,574 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=63, resume processing ppid=62 2023-03-29 03:56:50,574 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=63, ppid=62, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=94b16ca160750195aa137348cdb80ba9, UNASSIGN in 1.9210 sec 2023-03-29 03:56:50,574 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680062210574"}]},"ts":"1680062210574"} 2023-03-29 03:56:50,575 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-03-29 03:56:50,576 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-03-29 03:56:50,577 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=62, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.9320 sec 2023-03-29 03:56:50,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-03-29 03:56:50,751 INFO [Listener at localhost/35027] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 62 completed 2023-03-29 03:56:50,751 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-03-29 03:56:50,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] procedure2.ProcedureExecutor(1029): Stored pid=65, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-03-29 03:56:50,753 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=65, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-29 03:56:50,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=65 2023-03-29 03:56:50,754 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=65, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-29 03:56:50,759 DEBUG [HFileArchiver-13] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/.tmp/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9 2023-03-29 03:56:50,761 DEBUG [HFileArchiver-13] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/.tmp/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A, FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/.tmp/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B, FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/.tmp/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C, FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/.tmp/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/recovered.edits] 2023-03-29 03:56:50,763 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/.tmp/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/91e2346552f341af9aabad616a4a6349 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/91e2346552f341af9aabad616a4a6349 2023-03-29 03:56:50,764 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/.tmp/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/d39c77ddc4f3415f960ef4daf450e6f6 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/d39c77ddc4f3415f960ef4daf450e6f6 2023-03-29 03:56:50,765 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/.tmp/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/d950cb46e2d74504ba633169be4a3e74 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/A/d950cb46e2d74504ba633169be4a3e74 2023-03-29 03:56:50,767 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/.tmp/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/0a706e94567548c4a2c9eff11ed2009f to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/0a706e94567548c4a2c9eff11ed2009f 2023-03-29 03:56:50,767 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/.tmp/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/45116042b05141d8a6277943ebe47c56 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/45116042b05141d8a6277943ebe47c56 2023-03-29 03:56:50,768 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/.tmp/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/8e788225d4fc46f59e082824254224b9 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/B/8e788225d4fc46f59e082824254224b9 2023-03-29 03:56:50,770 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/.tmp/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/297cd7dd068e4df6b4941c8e4a296a0a to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/297cd7dd068e4df6b4941c8e4a296a0a 2023-03-29 03:56:50,770 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/.tmp/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/ac98ebe871c248619215c94276b44641 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/C/ac98ebe871c248619215c94276b44641 2023-03-29 03:56:50,772 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/.tmp/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/recovered.edits/748.seqid to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9/recovered.edits/748.seqid 2023-03-29 03:56:50,773 DEBUG [HFileArchiver-13] backup.HFileArchiver(596): Deleted hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/.tmp/data/default/TestAcidGuarantees/94b16ca160750195aa137348cdb80ba9 2023-03-29 03:56:50,773 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-29 03:56:50,773 DEBUG [PEWorker-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-03-29 03:56:50,774 DEBUG [PEWorker-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-03-29 03:56:50,776 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303290a70b3f16d4e4bcc995365818ef58b83 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303290a70b3f16d4e4bcc995365818ef58b83 2023-03-29 03:56:50,777 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032914f6dcca863642bf9443f6d0bbc51c6e to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032914f6dcca863642bf9443f6d0bbc51c6e 2023-03-29 03:56:50,778 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303291f907ff0f69741b5a456b6c8794aa658 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303291f907ff0f69741b5a456b6c8794aa658 2023-03-29 03:56:50,778 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303291f9121152c5546c08f558b74e018bf4d to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303291f9121152c5546c08f558b74e018bf4d 2023-03-29 03:56:50,779 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032936885d881e34427883d258fac6b57f56 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032936885d881e34427883d258fac6b57f56 2023-03-29 03:56:50,780 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303293b5d44b2889e49f6b0d4985b4cb11c29 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303293b5d44b2889e49f6b0d4985b4cb11c29 2023-03-29 03:56:50,781 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303294c3cadbb5d704bfd8537c9497f039336 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303294c3cadbb5d704bfd8537c9497f039336 2023-03-29 03:56:50,782 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303295b60a75f385a40e78fdaee0ff6c6d1bc to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303295b60a75f385a40e78fdaee0ff6c6d1bc 2023-03-29 03:56:50,782 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032960b691a4c8404fed80aa2b6db0e3fcc2 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032960b691a4c8404fed80aa2b6db0e3fcc2 2023-03-29 03:56:50,783 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032960c727d994e14b36adbb57db96a8cd9d to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032960c727d994e14b36adbb57db96a8cd9d 2023-03-29 03:56:50,784 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032966df0757043f41ea8d36b2459ec11471 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032966df0757043f41ea8d36b2459ec11471 2023-03-29 03:56:50,785 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032986cc729bf52f46f8b8c3d74f1b4a7e1c to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032986cc729bf52f46f8b8c3d74f1b4a7e1c 2023-03-29 03:56:50,785 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032990a2574918bf476da1fdb97ecfe74634 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032990a2574918bf476da1fdb97ecfe74634 2023-03-29 03:56:50,786 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032990cbaa7c3659477f8336e27abcf91689 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032990cbaa7c3659477f8336e27abcf91689 2023-03-29 03:56:50,787 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230329920c94fb8f5543509b8cdd31b1dcef05 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230329920c94fb8f5543509b8cdd31b1dcef05 2023-03-29 03:56:50,788 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230329957670930d8843c5a8b52553f22d629e to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230329957670930d8843c5a8b52553f22d629e 2023-03-29 03:56:50,788 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032999e975310e0f4eb08355fdecf72eeb98 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032999e975310e0f4eb08355fdecf72eeb98 2023-03-29 03:56:50,789 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303299f730dc0ee3f4fbbb4cc51e2f705390a to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303299f730dc0ee3f4fbbb4cc51e2f705390a 2023-03-29 03:56:50,790 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230329a2ec5d0b939d4c43b300d3f2680d4fa4 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230329a2ec5d0b939d4c43b300d3f2680d4fa4 2023-03-29 03:56:50,791 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230329a4fe157ea7984e938c3f47c9ef34c8da to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230329a4fe157ea7984e938c3f47c9ef34c8da 2023-03-29 03:56:50,791 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230329a9b828dc0cfb4ecc9d87b23b1ed6ca47 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230329a9b828dc0cfb4ecc9d87b23b1ed6ca47 2023-03-29 03:56:50,792 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230329ba149981e56348e2ab0378ef72dd1f24 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230329ba149981e56348e2ab0378ef72dd1f24 2023-03-29 03:56:50,793 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230329c3323a0b271f4d24825c77a272145ee9 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230329c3323a0b271f4d24825c77a272145ee9 2023-03-29 03:56:50,794 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230329d5607cbb334b426bb95a7e98350b3ed8 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230329d5607cbb334b426bb95a7e98350b3ed8 2023-03-29 03:56:50,795 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230329d9796be8888b4f2ba1f3ad4ee620c2b3 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230329d9796be8888b4f2ba1f3ad4ee620c2b3 2023-03-29 03:56:50,796 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230329dc98d24f26d74e0abd2d90f69bbbb61c to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230329dc98d24f26d74e0abd2d90f69bbbb61c 2023-03-29 03:56:50,797 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230329df6580e982b5479195677a2a28fc165c to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230329df6580e982b5479195677a2a28fc165c 2023-03-29 03:56:50,798 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230329e26150dc8740489f8769c6526c37f590 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230329e26150dc8740489f8769c6526c37f590 2023-03-29 03:56:50,799 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230329ebf03b4f56ea401584ef8432a8d06e1a to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230329ebf03b4f56ea401584ef8432a8d06e1a 2023-03-29 03:56:50,800 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230329ed1c52eb594640c4a98637156b07857d to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230329ed1c52eb594640c4a98637156b07857d 2023-03-29 03:56:50,801 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230329f7525d37bc264c62aebd7132e24f9e17 to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230329f7525d37bc264c62aebd7132e24f9e17 2023-03-29 03:56:50,801 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230329fc7b33d62b3e47f3a2771625e7c3801b to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230329fc7b33d62b3e47f3a2771625e7c3801b 2023-03-29 03:56:50,802 DEBUG [PEWorker-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230329fe00076e78f840bf8a0fc1e20b1d67cd to hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230329fe00076e78f840bf8a0fc1e20b1d67cd 2023-03-29 03:56:50,802 DEBUG [PEWorker-3] backup.HFileArchiver(596): Deleted hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-03-29 03:56:50,804 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=65, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-29 03:56:50,806 WARN [PEWorker-3] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-03-29 03:56:50,807 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-03-29 03:56:50,808 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=65, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-29 03:56:50,808 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-03-29 03:56:50,808 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1680062210808"}]},"ts":"9223372036854775807"} 2023-03-29 03:56:50,809 INFO [PEWorker-3] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-03-29 03:56:50,809 DEBUG [PEWorker-3] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 94b16ca160750195aa137348cdb80ba9, NAME => 'TestAcidGuarantees,,1680062176940.94b16ca160750195aa137348cdb80ba9.', STARTKEY => '', ENDKEY => ''}] 2023-03-29 03:56:50,809 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-03-29 03:56:50,810 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1680062210810"}]},"ts":"9223372036854775807"} 2023-03-29 03:56:50,811 INFO [PEWorker-3] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-03-29 03:56:50,812 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(130): Finished pid=65, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-29 03:56:50,813 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=65, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 61 msec 2023-03-29 03:56:50,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43505] master.MasterRpcServices(1227): Checking to see if procedure is done pid=65 2023-03-29 03:56:50,855 INFO [Listener at localhost/35027] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 65 completed 2023-03-29 03:56:50,866 INFO [Listener at localhost/35027] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobGetAtomicity Thread=225 (was 225), OpenFileDescriptor=556 (was 567), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=358 (was 392), ProcessCount=169 (was 169), AvailableMemoryMB=7010 (was 6984) - AvailableMemoryMB LEAK? - 2023-03-29 03:56:50,866 INFO [Listener at localhost/35027] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-03-29 03:56:50,867 INFO [Listener at localhost/35027] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-03-29 03:56:50,867 DEBUG [Listener at localhost/35027] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3ddd91d3 to 127.0.0.1:52103 2023-03-29 03:56:50,867 DEBUG [Listener at localhost/35027] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-29 03:56:50,867 DEBUG [Listener at localhost/35027] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-03-29 03:56:50,867 DEBUG [Listener at localhost/35027] util.JVMClusterUtil(257): Found active master hash=1764192222, stopped=false 2023-03-29 03:56:50,867 INFO [Listener at localhost/35027] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase4.apache.org,43505,1680062017306 2023-03-29 03:56:50,869 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-03-29 03:56:50,869 INFO [Listener at localhost/35027] procedure2.ProcedureExecutor(629): Stopping 2023-03-29 03:56:50,869 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-03-29 03:56:50,869 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-29 03:56:50,869 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-03-29 03:56:50,869 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-03-29 03:56:50,869 DEBUG [Listener at localhost/35027] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5465106c to 127.0.0.1:52103 2023-03-29 03:56:50,870 DEBUG [Listener at localhost/35027] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-29 03:56:50,870 INFO [Listener at localhost/35027] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,35599,1680062018534' ***** 2023-03-29 03:56:50,870 INFO [Listener at localhost/35027] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-03-29 03:56:50,870 INFO [RS:0;jenkins-hbase4:35599] regionserver.HeapMemoryManager(220): Stopping 2023-03-29 03:56:50,870 INFO [RS:0;jenkins-hbase4:35599] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-03-29 03:56:50,870 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-03-29 03:56:50,870 INFO [RS:0;jenkins-hbase4:35599] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-03-29 03:56:50,871 INFO [RS:0;jenkins-hbase4:35599] regionserver.HRegionServer(3304): Received CLOSE for 3fa241e8c8920ab38706390b2493ecb8 2023-03-29 03:56:50,871 INFO [RS:0;jenkins-hbase4:35599] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:50,871 DEBUG [RS:0;jenkins-hbase4:35599] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1a0467bd to 127.0.0.1:52103 2023-03-29 03:56:50,872 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 3fa241e8c8920ab38706390b2493ecb8, disabling compactions & flushes 2023-03-29 03:56:50,872 DEBUG [RS:0;jenkins-hbase4:35599] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-29 03:56:50,872 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1680062020474.3fa241e8c8920ab38706390b2493ecb8. 2023-03-29 03:56:50,872 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1680062020474.3fa241e8c8920ab38706390b2493ecb8. 2023-03-29 03:56:50,872 INFO [RS:0;jenkins-hbase4:35599] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-03-29 03:56:50,872 INFO [RS:0;jenkins-hbase4:35599] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-03-29 03:56:50,872 INFO [RS:0;jenkins-hbase4:35599] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-03-29 03:56:50,872 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1680062020474.3fa241e8c8920ab38706390b2493ecb8. after waiting 0 ms 2023-03-29 03:56:50,872 INFO [RS:0;jenkins-hbase4:35599] regionserver.HRegionServer(3304): Received CLOSE for 1588230740 2023-03-29 03:56:50,872 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1680062020474.3fa241e8c8920ab38706390b2493ecb8. 2023-03-29 03:56:50,872 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 3fa241e8c8920ab38706390b2493ecb8 1/1 column families, dataSize=78 B heapSize=488 B 2023-03-29 03:56:50,872 INFO [RS:0;jenkins-hbase4:35599] regionserver.HRegionServer(1475): Waiting on 2 regions to close 2023-03-29 03:56:50,872 DEBUG [RS:0;jenkins-hbase4:35599] regionserver.HRegionServer(1479): Online Regions={1588230740=hbase:meta,,1.1588230740, 3fa241e8c8920ab38706390b2493ecb8=hbase:namespace,,1680062020474.3fa241e8c8920ab38706390b2493ecb8.} 2023-03-29 03:56:50,873 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-03-29 03:56:50,873 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-03-29 03:56:50,873 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-03-29 03:56:50,873 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-03-29 03:56:50,873 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-03-29 03:56:50,873 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=20.92 KB heapSize=36.24 KB 2023-03-29 03:56:50,874 DEBUG [RS:0;jenkins-hbase4:35599] regionserver.HRegionServer(1505): Waiting on 1588230740, 3fa241e8c8920ab38706390b2493ecb8 2023-03-29 03:56:50,885 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=18.26 KB at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/hbase/meta/1588230740/.tmp/info/b05ae9d86c9e49a9a48ea5694006c5dc 2023-03-29 03:56:50,889 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for b05ae9d86c9e49a9a48ea5694006c5dc 2023-03-29 03:56:50,957 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-03-29 03:56:51,074 DEBUG [RS:0;jenkins-hbase4:35599] regionserver.HRegionServer(1505): Waiting on 1588230740, 3fa241e8c8920ab38706390b2493ecb8 2023-03-29 03:56:51,275 DEBUG [RS:0;jenkins-hbase4:35599] regionserver.HRegionServer(1505): Waiting on 1588230740, 3fa241e8c8920ab38706390b2493ecb8 2023-03-29 03:56:51,286 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/hbase/namespace/3fa241e8c8920ab38706390b2493ecb8/.tmp/info/4c9a55b0979f477e82afa95131cbb5cf 2023-03-29 03:56:51,291 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/hbase/namespace/3fa241e8c8920ab38706390b2493ecb8/.tmp/info/4c9a55b0979f477e82afa95131cbb5cf as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/hbase/namespace/3fa241e8c8920ab38706390b2493ecb8/info/4c9a55b0979f477e82afa95131cbb5cf 2023-03-29 03:56:51,294 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/hbase/namespace/3fa241e8c8920ab38706390b2493ecb8/info/4c9a55b0979f477e82afa95131cbb5cf, entries=2, sequenceid=6, filesize=4.8 K 2023-03-29 03:56:51,295 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=588 B at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/hbase/meta/1588230740/.tmp/rep_barrier/63764be77b1248c994af3bcf8ed26314 2023-03-29 03:56:51,295 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for 3fa241e8c8920ab38706390b2493ecb8 in 423ms, sequenceid=6, compaction requested=false 2023-03-29 03:56:51,298 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/hbase/namespace/3fa241e8c8920ab38706390b2493ecb8/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=1 2023-03-29 03:56:51,298 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 63764be77b1248c994af3bcf8ed26314 2023-03-29 03:56:51,299 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1680062020474.3fa241e8c8920ab38706390b2493ecb8. 2023-03-29 03:56:51,299 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 3fa241e8c8920ab38706390b2493ecb8: 2023-03-29 03:56:51,300 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1680062020474.3fa241e8c8920ab38706390b2493ecb8. 2023-03-29 03:56:51,475 DEBUG [RS:0;jenkins-hbase4:35599] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-03-29 03:56:51,675 DEBUG [RS:0;jenkins-hbase4:35599] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-03-29 03:56:51,708 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.08 KB at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/hbase/meta/1588230740/.tmp/table/8f118ba9346e4042b152e00157fe95da 2023-03-29 03:56:51,711 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 8f118ba9346e4042b152e00157fe95da 2023-03-29 03:56:51,711 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/hbase/meta/1588230740/.tmp/info/b05ae9d86c9e49a9a48ea5694006c5dc as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/hbase/meta/1588230740/info/b05ae9d86c9e49a9a48ea5694006c5dc 2023-03-29 03:56:51,714 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for b05ae9d86c9e49a9a48ea5694006c5dc 2023-03-29 03:56:51,714 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/hbase/meta/1588230740/info/b05ae9d86c9e49a9a48ea5694006c5dc, entries=22, sequenceid=93, filesize=7.1 K 2023-03-29 03:56:51,716 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/hbase/meta/1588230740/.tmp/rep_barrier/63764be77b1248c994af3bcf8ed26314 as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/hbase/meta/1588230740/rep_barrier/63764be77b1248c994af3bcf8ed26314 2023-03-29 03:56:51,718 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 63764be77b1248c994af3bcf8ed26314 2023-03-29 03:56:51,718 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/hbase/meta/1588230740/rep_barrier/63764be77b1248c994af3bcf8ed26314, entries=6, sequenceid=93, filesize=5.5 K 2023-03-29 03:56:51,718 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/hbase/meta/1588230740/.tmp/table/8f118ba9346e4042b152e00157fe95da as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/hbase/meta/1588230740/table/8f118ba9346e4042b152e00157fe95da 2023-03-29 03:56:51,721 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 8f118ba9346e4042b152e00157fe95da 2023-03-29 03:56:51,721 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/hbase/meta/1588230740/table/8f118ba9346e4042b152e00157fe95da, entries=9, sequenceid=93, filesize=5.5 K 2023-03-29 03:56:51,722 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.92 KB/21417, heapSize ~36.20 KB/37064, currentSize=0 B/0 for 1588230740 in 849ms, sequenceid=93, compaction requested=false 2023-03-29 03:56:51,725 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/data/hbase/meta/1588230740/recovered.edits/96.seqid, newMaxSeqId=96, maxSeqId=1 2023-03-29 03:56:51,726 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-03-29 03:56:51,727 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-03-29 03:56:51,727 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-03-29 03:56:51,727 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-03-29 03:56:51,875 INFO [RS:0;jenkins-hbase4:35599] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,35599,1680062018534; all regions closed. 2023-03-29 03:56:51,883 DEBUG [RS:0;jenkins-hbase4:35599] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/oldWALs 2023-03-29 03:56:51,883 INFO [RS:0;jenkins-hbase4:35599] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C35599%2C1680062018534.meta:.meta(num 1680062020198) 2023-03-29 03:56:51,887 DEBUG [RS:0;jenkins-hbase4:35599] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/oldWALs 2023-03-29 03:56:51,887 INFO [RS:0;jenkins-hbase4:35599] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C35599%2C1680062018534:(num 1680062020015) 2023-03-29 03:56:51,887 DEBUG [RS:0;jenkins-hbase4:35599] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-29 03:56:51,888 INFO [RS:0;jenkins-hbase4:35599] regionserver.LeaseManager(133): Closed leases 2023-03-29 03:56:51,888 INFO [RS:0;jenkins-hbase4:35599] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-03-29 03:56:51,888 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-03-29 03:56:51,890 INFO [RS:0;jenkins-hbase4:35599] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:35599 2023-03-29 03:56:51,895 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,35599,1680062018534 2023-03-29 03:56:51,895 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-03-29 03:56:51,895 ERROR [Listener at localhost/35027-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@3059765d rejected from java.util.concurrent.ThreadPoolExecutor@3466e48e[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2766] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-03-29 03:56:51,896 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-03-29 03:56:51,896 ERROR [Listener at localhost/35027-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@3b4ffccb rejected from java.util.concurrent.ThreadPoolExecutor@3466e48e[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2766] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-03-29 03:56:51,897 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,35599,1680062018534] 2023-03-29 03:56:51,897 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,35599,1680062018534; numProcessing=1 2023-03-29 03:56:51,899 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,35599,1680062018534 already deleted, retry=false 2023-03-29 03:56:51,899 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,35599,1680062018534 expired; onlineServers=0 2023-03-29 03:56:51,899 INFO [RegionServerTracker-0] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,43505,1680062017306' ***** 2023-03-29 03:56:51,899 INFO [RegionServerTracker-0] regionserver.HRegionServer(2310): STOPPED: Cluster shutdown set; onlineServer=0 2023-03-29 03:56:51,899 DEBUG [M:0;jenkins-hbase4:43505] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@281a65e5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-03-29 03:56:51,899 INFO [M:0;jenkins-hbase4:43505] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,43505,1680062017306 2023-03-29 03:56:51,899 INFO [M:0;jenkins-hbase4:43505] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,43505,1680062017306; all regions closed. 2023-03-29 03:56:51,899 DEBUG [M:0;jenkins-hbase4:43505] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-29 03:56:51,899 DEBUG [M:0;jenkins-hbase4:43505] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-03-29 03:56:51,899 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-03-29 03:56:51,899 DEBUG [M:0;jenkins-hbase4:43505] cleaner.HFileCleaner(317): Stopping file delete threads 2023-03-29 03:56:51,899 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1680062019671] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1680062019671,5,FailOnTimeoutGroup] 2023-03-29 03:56:51,899 INFO [M:0;jenkins-hbase4:43505] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-03-29 03:56:51,899 INFO [M:0;jenkins-hbase4:43505] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-03-29 03:56:51,899 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1680062019672] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1680062019672,5,FailOnTimeoutGroup] 2023-03-29 03:56:51,900 INFO [M:0;jenkins-hbase4:43505] hbase.ChoreService(369): Chore service for: master/jenkins-hbase4:0 had [] on shutdown 2023-03-29 03:56:51,900 DEBUG [M:0;jenkins-hbase4:43505] master.HMaster(1512): Stopping service threads 2023-03-29 03:56:51,900 INFO [M:0;jenkins-hbase4:43505] procedure2.RemoteProcedureDispatcher(118): Stopping procedure remote dispatcher 2023-03-29 03:56:51,900 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-03-29 03:56:51,900 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-29 03:56:51,900 ERROR [M:0;jenkins-hbase4:43505] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[IPC Client (791558932) connection to localhost/127.0.0.1:42373 from jenkins,5,PEWorkerGroup] Thread[HFileArchiver-13,5,PEWorkerGroup] 2023-03-29 03:56:51,901 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-03-29 03:56:51,901 INFO [M:0;jenkins-hbase4:43505] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-03-29 03:56:51,901 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-03-29 03:56:51,901 DEBUG [M:0;jenkins-hbase4:43505] zookeeper.ZKUtil(398): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-03-29 03:56:51,902 WARN [M:0;jenkins-hbase4:43505] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-03-29 03:56:51,902 INFO [M:0;jenkins-hbase4:43505] assignment.AssignmentManager(315): Stopping assignment manager 2023-03-29 03:56:51,902 INFO [M:0;jenkins-hbase4:43505] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-03-29 03:56:51,902 DEBUG [M:0;jenkins-hbase4:43505] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-03-29 03:56:51,902 INFO [M:0;jenkins-hbase4:43505] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-29 03:56:51,902 DEBUG [M:0;jenkins-hbase4:43505] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-29 03:56:51,903 DEBUG [M:0;jenkins-hbase4:43505] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-03-29 03:56:51,903 DEBUG [M:0;jenkins-hbase4:43505] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-29 03:56:51,903 INFO [M:0;jenkins-hbase4:43505] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=344.85 KB heapSize=393.55 KB 2023-03-29 03:56:51,998 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-03-29 03:56:51,998 INFO [RS:0;jenkins-hbase4:35599] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,35599,1680062018534; zookeeper connection closed. 2023-03-29 03:56:51,998 ERROR [Listener at localhost/35027-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@431e5060 rejected from java.util.concurrent.ThreadPoolExecutor@3466e48e[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2766] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-03-29 03:56:51,998 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): regionserver:35599-0x1012e67d5710001, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-03-29 03:56:51,998 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@6ae31326] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@6ae31326 2023-03-29 03:56:51,998 ERROR [Listener at localhost/35027-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@55a68c6f rejected from java.util.concurrent.ThreadPoolExecutor@3466e48e[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2766] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-03-29 03:56:52,000 INFO [Listener at localhost/35027] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 1 regionserver(s) complete 2023-03-29 03:56:52,323 INFO [M:0;jenkins-hbase4:43505] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=344.85 KB at sequenceid=578 (bloomFilter=true), to=hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/cd8a1a03885a4af69ffb4ef7e95671de 2023-03-29 03:56:52,326 INFO [M:0;jenkins-hbase4:43505] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for cd8a1a03885a4af69ffb4ef7e95671de 2023-03-29 03:56:52,326 DEBUG [M:0;jenkins-hbase4:43505] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/cd8a1a03885a4af69ffb4ef7e95671de as hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/cd8a1a03885a4af69ffb4ef7e95671de 2023-03-29 03:56:52,329 INFO [M:0;jenkins-hbase4:43505] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for cd8a1a03885a4af69ffb4ef7e95671de 2023-03-29 03:56:52,329 INFO [M:0;jenkins-hbase4:43505] regionserver.HStore(1080): Added hdfs://localhost:42373/user/jenkins/test-data/7ac5a585-7577-319e-f41c-93537a3d416f/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/cd8a1a03885a4af69ffb4ef7e95671de, entries=65, sequenceid=578, filesize=24.7 K 2023-03-29 03:56:52,330 INFO [M:0;jenkins-hbase4:43505] regionserver.HRegion(2948): Finished flush of dataSize ~344.85 KB/353126, heapSize ~393.53 KB/402976, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 427ms, sequenceid=578, compaction requested=false 2023-03-29 03:56:52,330 INFO [M:0;jenkins-hbase4:43505] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-29 03:56:52,330 DEBUG [M:0;jenkins-hbase4:43505] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-03-29 03:56:52,333 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-03-29 03:56:52,333 INFO [M:0;jenkins-hbase4:43505] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-03-29 03:56:52,334 INFO [M:0;jenkins-hbase4:43505] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:43505 2023-03-29 03:56:52,335 DEBUG [M:0;jenkins-hbase4:43505] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase4.apache.org,43505,1680062017306 already deleted, retry=false 2023-03-29 03:56:52,438 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-03-29 03:56:52,438 INFO [M:0;jenkins-hbase4:43505] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,43505,1680062017306; zookeeper connection closed. 2023-03-29 03:56:52,438 ERROR [Listener at localhost/35027-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@5dddb59c rejected from java.util.concurrent.ThreadPoolExecutor@3af49ebc[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4620] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-03-29 03:56:52,438 DEBUG [Listener at localhost/35027-EventThread] zookeeper.ZKWatcher(600): master:43505-0x1012e67d5710000, quorum=127.0.0.1:52103, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-03-29 03:56:52,438 ERROR [Listener at localhost/35027-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@faefaea rejected from java.util.concurrent.ThreadPoolExecutor@3af49ebc[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4620] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-03-29 03:56:52,440 WARN [Listener at localhost/35027] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-03-29 03:56:52,443 INFO [Listener at localhost/35027] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-03-29 03:56:52,554 WARN [BP-347134095-172.31.14.131-1680062013777 heartbeating to localhost/127.0.0.1:42373] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-03-29 03:56:52,554 WARN [BP-347134095-172.31.14.131-1680062013777 heartbeating to localhost/127.0.0.1:42373] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-347134095-172.31.14.131-1680062013777 (Datanode Uuid ff5b43e4-08a3-49e1-bbac-436ca110ab7a) service to localhost/127.0.0.1:42373 2023-03-29 03:56:52,558 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/3541dc51-7937-e96e-6c09-468c657fb2cc/cluster_c6f3e91d-ff3e-c1b8-1e7d-cfb1945e516d/dfs/data/data1/current/BP-347134095-172.31.14.131-1680062013777] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-03-29 03:56:52,559 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/3541dc51-7937-e96e-6c09-468c657fb2cc/cluster_c6f3e91d-ff3e-c1b8-1e7d-cfb1945e516d/dfs/data/data2/current/BP-347134095-172.31.14.131-1680062013777] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-03-29 03:56:52,596 INFO [Listener at localhost/35027] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-03-29 03:56:52,770 INFO [Listener at localhost/35027] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-03-29 03:56:52,953 INFO [Listener at localhost/35027] hbase.HBaseTestingUtility(1293): Minicluster is down